Nov 03 18:41:44.752693 ip-10-0-133-236 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 18:41:44.752702 ip-10-0-133-236 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 18:41:44.752710 ip-10-0-133-236 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 18:41:44.752929 ip-10-0-133-236 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 18:41:54.797407 ip-10-0-133-236 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 18:41:54.797424 ip-10-0-133-236 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot aa209c8a993a4ffe8ffe31c1566c39f2 -- Nov 03 18:44:05.877405 ip-10-0-133-236 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:44:06.253886 ip-10-0-133-236 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:06.253886 ip-10-0-133-236 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:44:06.253886 ip-10-0-133-236 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:06.253886 ip-10-0-133-236 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:06.257237 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257157 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.257237 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257236 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.257237 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257240 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257243 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257246 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257249 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257252 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257255 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257258 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257260 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257263 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257266 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257269 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257271 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257274 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257277 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257279 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257282 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257285 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257293 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257296 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257298 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.257330 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257301 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257304 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257306 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257309 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257311 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257314 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257317 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257319 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257322 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257324 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257327 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257330 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257333 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257336 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257339 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257342 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257344 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257347 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257350 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.257792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257353 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257355 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257358 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257361 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257363 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257366 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257368 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257371 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257374 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257376 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257379 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257381 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257384 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257386 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257389 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257393 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257396 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257398 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257401 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.258287 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257403 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257406 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257408 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257411 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257413 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257416 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257419 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257421 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257425 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257427 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257430 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257432 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257436 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257440 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257444 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257447 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257450 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257452 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257455 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.258740 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257458 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257460 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257463 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257466 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257468 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257471 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257473 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257476 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257479 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257482 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257485 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257488 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257490 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257493 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257496 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257499 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257501 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257504 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257507 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257509 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257512 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.259199 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257514 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257517 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257522 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257526 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257528 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.257533 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259590 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259598 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259601 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259604 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259606 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259609 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259612 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259615 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259617 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259620 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259623 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259626 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259628 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.259683 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259631 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259635 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259639 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259642 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259645 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259648 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259650 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259653 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259655 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259658 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259661 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259663 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259666 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259668 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259672 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259675 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259678 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259681 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259683 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.260146 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259687 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259690 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259693 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259696 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259698 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259701 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259703 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259706 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259709 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259711 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259714 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259716 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259719 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259721 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259724 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259728 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259732 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259735 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259738 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.260594 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259741 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259743 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259746 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259749 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259751 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259754 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259756 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259759 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259762 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259769 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259771 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259774 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259776 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259780 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259782 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259785 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259788 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259790 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.261162 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259792 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259795 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259798 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259800 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259803 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259805 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259808 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259811 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259813 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259816 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259818 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259821 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259824 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259827 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259830 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259833 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259835 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259838 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259840 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259845 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.261619 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259848 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259850 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259853 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259856 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259861 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259864 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259866 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259869 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259872 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259874 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259877 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259879 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259882 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259884 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.259888 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260375 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260380 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260384 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260387 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260390 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.262100 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260393 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260395 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260400 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260404 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260408 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260411 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260414 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260417 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260419 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260422 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260425 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260427 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260430 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260432 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260434 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260437 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260440 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260442 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260445 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.262621 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260451 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260454 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260457 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260461 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260464 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260467 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260471 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260474 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260477 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260479 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260482 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260485 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260488 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260491 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260493 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260496 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260499 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260501 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260504 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.263078 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260507 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260509 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260512 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260514 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260517 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260520 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260522 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260525 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260527 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260530 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260533 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260535 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260538 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260540 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260544 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260547 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260549 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260552 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260554 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260558 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.263579 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260560 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260563 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260565 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260567 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260570 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260573 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260576 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260578 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260581 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260583 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260586 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260588 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260591 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260593 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260596 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260599 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260601 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260604 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260607 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260610 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.264060 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260612 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260615 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260617 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260620 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260622 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260625 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260628 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260630 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260635 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260638 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260641 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260644 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260647 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260649 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260652 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260654 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260657 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260659 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260662 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260665 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.264545 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260668 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260671 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.260673 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260795 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260804 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260810 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260814 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260819 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260822 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260826 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260830 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260834 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260837 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260840 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260843 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260846 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260849 2580 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260852 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260857 2580 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260861 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260867 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260872 2580 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260875 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260878 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:44:06.265069 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260881 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260885 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260889 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260892 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260895 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260898 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260901 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260905 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260908 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260911 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260914 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260918 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260921 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260924 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260927 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260931 2580 flags.go:64] FLAG: --enable-server="true" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260933 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260939 2580 flags.go:64] FLAG: --event-burst="100" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260942 2580 flags.go:64] FLAG: --event-qps="50" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260945 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260948 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260951 2580 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260955 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260959 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260962 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:44:06.265646 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260965 2580 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260970 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260973 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260977 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260981 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260984 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260987 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260990 2580 flags.go:64] FLAG: --feature-gates="" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260994 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.260997 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261000 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261003 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261007 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261009 2580 flags.go:64] FLAG: --help="false" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261012 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261016 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261019 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261030 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261033 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261037 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261040 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261043 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261046 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:44:06.266283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261049 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261052 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261055 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261058 2580 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261061 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261064 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261067 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261070 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261073 2580 flags.go:64] FLAG: --lock-file="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261076 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261080 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261083 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261090 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261093 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261097 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261099 2580 flags.go:64] FLAG: --logging-format="text" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261102 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261106 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261109 2580 flags.go:64] FLAG: --manifest-url="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261112 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261127 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261130 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261135 2580 flags.go:64] FLAG: --max-pods="110" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261138 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261141 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:44:06.266837 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261144 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261147 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261150 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261153 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261156 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261163 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261166 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261169 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261172 2580 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261175 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261178 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261181 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261184 2580 flags.go:64] FLAG: --port="10250" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261187 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261190 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0907cfa58e4a560e9" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261194 2580 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261196 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261201 2580 flags.go:64] FLAG: --register-node="true" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261204 2580 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261209 2580 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261212 2580 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261215 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261219 2580 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261222 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261225 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261228 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:44:06.267474 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261231 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261234 2580 flags.go:64] FLAG: --runonce="false" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261237 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261240 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261243 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261246 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261249 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261252 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261255 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261258 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261261 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261264 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261267 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261270 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261273 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261276 2580 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261279 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261284 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261287 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261290 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261295 2580 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261297 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261300 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261304 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261307 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:44:06.268077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261312 2580 flags.go:64] FLAG: --v="2" Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261316 2580 flags.go:64] FLAG: --version="false" Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261320 2580 flags.go:64] FLAG: --vmodule="" Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261324 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261328 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261464 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261468 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261471 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261473 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261476 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261479 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261482 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261484 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261487 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261490 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261493 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261495 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261498 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261500 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261503 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.268707 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261506 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261508 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261511 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261514 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261517 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261530 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261533 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261536 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261539 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261542 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261545 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261548 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261552 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261555 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261557 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261560 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261563 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261566 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261568 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.269221 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261572 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261575 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261577 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261580 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261582 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261585 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261588 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261591 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261593 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261596 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261599 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261601 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261604 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261606 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261609 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261612 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261615 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261617 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261620 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261623 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.269689 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261625 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261628 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261631 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261635 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261638 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261642 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261645 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261648 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261651 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261653 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261656 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261659 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261662 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261664 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261667 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261670 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261672 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261675 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261678 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.270184 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261685 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261689 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261693 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261696 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261699 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261702 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261705 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261707 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261710 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261713 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261715 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261718 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261721 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261723 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261726 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261731 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261733 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261738 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.270878 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261740 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261743 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261746 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261748 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261751 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261754 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261757 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261761 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261765 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261769 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261772 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261775 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261778 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261781 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.261808 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261842 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.271450 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261846 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261849 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261852 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261854 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261857 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261860 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261862 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261865 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261868 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261870 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261873 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261875 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261878 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261882 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261884 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261887 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261890 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261893 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261896 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261898 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.271912 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261901 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261904 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261907 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261910 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261913 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261915 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261918 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261920 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261923 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261927 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261931 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261935 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261938 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261941 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261943 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261946 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261949 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261952 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261954 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.272464 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261957 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261959 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261962 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261964 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261967 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261969 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261974 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261978 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261980 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261985 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261988 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261990 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261993 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261997 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.261999 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262002 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262006 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262010 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262013 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262016 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.272910 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262018 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262021 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262024 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262026 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262029 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262031 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262034 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262039 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262042 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262044 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262047 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262049 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262052 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262055 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262057 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262060 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262062 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262066 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.273409 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262069 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262071 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262075 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262078 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262080 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262083 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262086 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262088 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262091 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262093 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262096 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262099 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262102 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262105 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262108 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262110 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262113 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262127 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262130 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.273845 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262132 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262135 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262138 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262140 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262143 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262146 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262148 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.262152 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.268564 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.268579 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.269688 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.269696 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269753 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269759 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269777 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269781 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269785 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269788 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269791 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.274339 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269793 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269796 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269799 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269802 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269804 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269807 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269810 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269812 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269815 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269817 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269820 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269823 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269826 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269831 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269834 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269836 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269839 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269841 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269844 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.274774 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269847 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269850 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269854 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269857 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269860 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269863 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269865 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269868 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269871 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269874 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269876 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269879 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269882 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269886 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269889 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269893 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269896 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269899 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269901 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269904 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.275324 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269907 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269909 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269912 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269914 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269917 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269920 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269923 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269925 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269928 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269931 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269933 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269936 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269939 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269942 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269945 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269948 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269951 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269954 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269956 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269959 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.275792 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269962 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269964 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269967 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269969 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269972 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269975 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269977 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269980 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269983 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269985 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269988 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269990 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269993 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269996 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.269999 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270001 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270004 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270007 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270009 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.276300 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270012 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270014 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270017 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270019 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270022 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270025 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270029 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270032 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270034 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270040 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270043 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270046 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270048 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270051 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270055 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270059 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270067 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270071 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.276730 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.270074 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.270105 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271629 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271648 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271674 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271694 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271775 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271781 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271785 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271796 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271801 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271805 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271809 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271813 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271817 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271821 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.277234 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271825 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271829 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271834 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271839 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271843 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271847 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271852 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271856 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271861 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271865 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271869 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271874 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271879 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271883 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271887 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271891 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271896 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271899 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271904 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271909 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.277633 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271914 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271918 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271924 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271928 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271932 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271936 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271941 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271945 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271949 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271953 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271957 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271961 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271965 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271970 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271975 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271979 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271984 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271987 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271992 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.278145 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.271996 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272000 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272005 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272009 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272016 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272020 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272030 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272035 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272042 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272045 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272049 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272051 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272054 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272058 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272063 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272066 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.278655 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272068 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272071 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272074 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272077 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272080 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272083 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272085 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272088 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272091 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272093 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272097 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272102 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272105 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272108 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272111 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272113 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272132 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272135 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272138 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.279189 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272140 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272143 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272146 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272149 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272152 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272155 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272158 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272161 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272163 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272166 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272169 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272171 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272174 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272177 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272180 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.272213 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:06.279626 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272251 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272255 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272258 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272261 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272263 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272266 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272269 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272272 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272275 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272278 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272282 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272284 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272287 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272290 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272293 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272295 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272298 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272301 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272303 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:06.280006 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272307 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272310 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272312 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272316 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272320 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272323 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272326 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272329 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272333 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272335 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272338 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272341 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272343 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272346 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272348 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272351 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272354 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272357 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272359 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:06.280490 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272362 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272365 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272367 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272370 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272373 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272375 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272379 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272383 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272386 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272389 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272392 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272395 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272398 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272401 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272404 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272407 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272410 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272413 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272415 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:06.280973 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272418 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272421 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272423 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272426 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272428 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272431 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272433 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272436 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272438 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272442 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272445 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272448 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272450 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272453 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272455 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272458 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272461 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272463 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272466 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272469 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:06.281555 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272472 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272475 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272477 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272480 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272483 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272485 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272488 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272490 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272493 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272496 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272498 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272501 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272504 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272506 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272509 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272511 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272514 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272517 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272520 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272522 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:06.282058 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272525 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272527 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272530 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272533 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272535 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272538 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272540 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272545 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:06.272547 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.272599 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.274346 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.275262 2580 server.go:1014] "Starting client certificate rotation" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.275363 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.275401 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.277131 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:44:06.282553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.281661 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:44:06.292493 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.292476 2580 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:44:06.295488 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.295472 2580 log.go:25] "Validated CRI v1 image API" Nov 03 18:44:06.296251 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.296229 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:44:06.299371 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.299347 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b011e213-6a53-4430-97c9-307d3a78024d:/dev/nvme0n1p4 d7b0a5ae-75d1-4816-9eb1-c0df023beac6:/dev/nvme0n1p3] Nov 03 18:44:06.299420 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.299372 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 18:44:06.304598 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.304482 2580 manager.go:217] Machine: {Timestamp:2026-11-03 18:44:06.303245937 +0000 UTC m=+0.316385965 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098082 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b696ce05d015ddc6f0c76e4e52a8e SystemUUID:ec2b696c-e05d-015d-dc6f-0c76e4e52a8e BootID:aa209c8a-993a-4ffe-8ffe-31c1566c39f2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cc:7d:16:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cc:7d:16:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:7a:e3:a7:dc:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:44:06.305027 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.305011 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:44:06.305165 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.305152 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:44:06.308003 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.307983 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:44:06.308156 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308004 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-236.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:44:06.308205 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308158 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:44:06.308205 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308166 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:44:06.308259 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308205 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:44:06.308259 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308227 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:44:06.308998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.308986 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:44:06.312596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.312584 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:44:06.312724 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.312714 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:44:06.313361 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313352 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:44:06.313394 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313368 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:44:06.313394 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313378 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:44:06.313394 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313390 2580 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:44:06.313511 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313415 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:44:06.313916 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.313901 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:44:06.317560 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.317534 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:44:06.322517 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.322504 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:44:06.323171 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323158 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:44:06.323205 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323190 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:44:06.323361 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323350 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:44:06.323407 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323366 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:44:06.323407 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323372 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:44:06.323407 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323378 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:44:06.323407 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323383 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:44:06.323407 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.323390 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:44:06.324102 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324094 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:44:06.324144 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324103 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:44:06.324144 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324110 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:44:06.324144 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324129 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:44:06.324144 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324139 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:44:06.324266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324147 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:44:06.324788 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324779 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:44:06.324788 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.324787 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:44:06.329161 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.329148 2580 server.go:1263] "Started kubelet" Nov 03 18:44:06.329293 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.329261 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:44:06.329526 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.329244 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:44:06.329591 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.329549 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:44:06.329888 ip-10-0-133-236 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:44:06.330797 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.330778 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:44:06.331925 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.331913 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:44:06.334536 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.334509 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 18:44:06.335694 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.335677 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:44:06.335785 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.335694 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:44:06.335844 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.335780 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:06.336328 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336283 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:44:06.336328 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336284 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:44:06.336328 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336308 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:44:06.336505 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336370 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:44:06.336505 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336380 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:44:06.336603 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336575 2580 factory.go:55] Registering systemd factory Nov 03 18:44:06.336603 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336597 2580 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:44:06.336861 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336846 2580 factory.go:153] Registering CRI-O factory Nov 03 18:44:06.336927 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336865 2580 factory.go:223] Registration of the crio container factory successfully Nov 03 18:44:06.336927 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336914 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:44:06.337044 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336933 2580 factory.go:103] Registering Raw factory Nov 03 18:44:06.337044 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.336947 2580 manager.go:1199] Started watching for new ooms in manager Nov 03 18:44:06.338797 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.338775 2580 manager.go:319] Starting recovery of all containers Nov 03 18:44:06.353502 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.353482 2580 manager.go:324] Recovery completed Nov 03 18:44:06.357865 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.357850 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.361035 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.359733 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.361103 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.361050 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.361103 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.361067 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.362497 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.362484 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.362566 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.362485 2580 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:44:06.362566 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.362515 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:44:06.362566 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.362535 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:44:06.363203 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.363186 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.363255 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.363217 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.363255 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.363232 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.364755 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.364740 2580 policy_none.go:50] "Start" Nov 03 18:44:06.364800 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.364762 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:44:06.364800 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.364777 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:44:06.366942 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.366930 2580 policy_none.go:44] "Start" Nov 03 18:44:06.399731 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.399701 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:44:06.402082 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402058 2580 manager.go:341] "Starting Device Plugin manager" Nov 03 18:44:06.402182 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.402146 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:44:06.402182 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402163 2580 server.go:87] "Starting device plugin registration server" Nov 03 18:44:06.402423 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402402 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:44:06.402514 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402422 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:44:06.402621 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402605 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:44:06.402736 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402682 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:44:06.402736 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.402690 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:44:06.403094 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.403075 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:44:06.403214 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.403136 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:06.455057 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.455025 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:44:06.456192 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.456171 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:44:06.456192 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.456192 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:44:06.456314 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.456219 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:44:06.456314 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.456250 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 18:44:06.471268 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.471242 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-236.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 18:44:06.471268 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.471256 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-236.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 18:44:06.472154 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.471254 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-236.us-west-2.compute.internal.18e49d3f30e4df3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-236.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-236.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:06.329106238 +0000 UTC m=+0.342246267,LastTimestamp:2026-11-03 18:44:06.329106238 +0000 UTC m=+0.342246267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-236.us-west-2.compute.internal,}" Nov 03 18:44:06.502794 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.502769 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.503574 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.503552 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.503668 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.503590 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.503668 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.503604 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.503668 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.503628 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.532548 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.532389 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-236.us-west-2.compute.internal.18e49d3f32cc15aa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-236.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-236.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-236.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:06.361036202 +0000 UTC m=+0.374176232,LastTimestamp:2026-11-03 18:44:06.361036202 +0000 UTC m=+0.374176232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-236.us-west-2.compute.internal,}" Nov 03 18:44:06.533159 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.533104 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 18:44:06.533159 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.533107 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 18:44:06.533276 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.533105 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-236.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 18:44:06.536613 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.536597 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:44:06.536660 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.536638 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbn94" Nov 03 18:44:06.556726 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.556700 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal"] Nov 03 18:44:06.556820 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.556771 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.557678 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.557663 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.557754 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.557692 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.557754 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.557713 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.559905 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.559893 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.560058 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.560131 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560081 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.560627 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560610 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.560695 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560639 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.560695 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560649 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.560695 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560610 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.560781 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560719 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.560781 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.560732 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.563344 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.563328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.563413 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.563354 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:06.564017 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.564002 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:06.564084 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.564035 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:06.564084 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.564048 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:06.599396 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.599375 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.602052 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.602036 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.620887 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.620868 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:06.627161 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.627148 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.627220 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.627165 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-236.us-west-2.compute.internal\": node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:06.737932 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.737907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.737932 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.737938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.738106 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.737956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.752262 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.752245 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:06.838518 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.838518 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.838518 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.838681 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838558 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.838681 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.838681 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.838566 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.852731 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.852574 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:06.901760 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.901740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.904898 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:06.904883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:06.952968 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:06.952938 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.053369 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.053341 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.154407 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.154335 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.255127 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.255097 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.276170 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.276150 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 18:44:07.276320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.276285 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:44:07.276436 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.276353 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3904a6b916d941eabd9d7e63c223c68-8c1f0edaac096962.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-236.us-west-2.compute.internal.18e49d3f32cca969\": read tcp 10.0.133.236:39080->3.131.114.228:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-236.us-west-2.compute.internal.18e49d3f32cca969 default 5194 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-236.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-236.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-236.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:06 +0000 UTC,LastTimestamp:2026-11-03 18:44:06.5577168 +0000 UTC m=+0.570856830,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-236.us-west-2.compute.internal,}" Nov 03 18:44:07.356426 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.356401 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.456744 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.456660 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.540832 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.540792 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:39:07 +0000 UTC" deadline="2028-08-16 09:14:36.257364628 +0000 UTC" Nov 03 18:44:07.540832 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.540826 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15638h30m28.716542241s" Nov 03 18:44:07.557754 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.557728 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.585553 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.585530 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:07.586605 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.586580 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:07.586711 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.586621 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:07.586711 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.586636 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:07.657874 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.657847 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.758504 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:07.758429 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:44:07.765470 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.765447 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:07.766625 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.766607 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:07.836313 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.836285 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:07.903718 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.903691 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:44:07.904719 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.904700 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:44:07.925265 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.925246 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:07.969174 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:07.969148 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:44:08.038856 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.038820 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683633c577f920552f48984254eada9.slice/crio-703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73 WatchSource:0}: Error finding container 703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73: Status 404 returned error can't find the container with id 703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73 Nov 03 18:44:08.044168 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.044152 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:44:08.101533 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.101511 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1266a5d4aae1fd29adb6e6003e98506.slice/crio-68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77 WatchSource:0}: Error finding container 68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77: Status 404 returned error can't find the container with id 68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77 Nov 03 18:44:08.316668 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.316596 2580 apiserver.go:52] "Watching apiserver" Nov 03 18:44:08.336856 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.336823 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 18:44:08.432057 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.432025 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:44:08.432387 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.432367 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x7248","kube-system/konnectivity-agent-xnctw","kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk","openshift-multus/multus-222xl","openshift-multus/multus-additional-cni-plugins-75kw7","openshift-cluster-node-tuning-operator/tuned-js4mm","openshift-dns/node-resolver-pxj4z","openshift-image-registry/node-ca-z9xv5","openshift-network-operator/iptables-alerter-dvjcd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-gvgvl","openshift-network-diagnostics/network-check-target-dbs68","openshift-ovn-kubernetes/ovnkube-node-ndlb6"] Nov 03 18:44:08.436117 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.436098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.438434 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.438406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:08.438556 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.438476 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:08.440791 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.440771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.443102 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.443085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.445343 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-222xl" Nov 03 18:44:08.445732 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445821 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445821 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445821 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445999 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445999 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445999 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445999 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.445999 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.445996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.446034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddbr\" (UniqueName: \"kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.446065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.446103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.446139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.446266 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.446174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.447761 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.447739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.450218 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.450197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.452663 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.452645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.452754 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.452699 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:08.456219 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.455832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.459392 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.459367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.461801 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.461777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.465165 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.465145 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.465277 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.465217 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:08.467478 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.467436 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerStarted","Data":"68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77"} Nov 03 18:44:08.467568 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.467492 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"0683633c577f920552f48984254eada9","Type":"ContainerStarted","Data":"703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73"} Nov 03 18:44:08.496893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.496873 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.496893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.496886 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.497006 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.496881 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\"" Nov 03 18:44:08.498838 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.498822 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.498907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.498877 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:44:08.499160 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.499148 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\"" Nov 03 18:44:08.499349 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.499334 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:44:08.537702 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.537665 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:44:08.541626 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541601 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:44:08.541626 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541623 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.541782 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541689 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.541782 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541725 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\"" Nov 03 18:44:08.541782 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541741 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:44:08.541907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541832 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\"" Nov 03 18:44:08.541907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541850 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:44:08.541907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541886 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:44:08.541907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541885 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.541907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541902 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.542083 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541910 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:44:08.542083 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541935 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.542083 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.541885 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\"" Nov 03 18:44:08.547159 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.547227 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547265 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.547307 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547307 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547383 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9cl\" (UniqueName: \"kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547383 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547308 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547383 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.547482 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547482 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.547482 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.547582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.547582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547498 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.547757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.547757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq8w\" (UniqueName: \"kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547782 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\"" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547791 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.547931 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547962 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.547981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddbr\" (UniqueName: \"kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548041 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\"" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjrt\" (UniqueName: \"kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtlq\" (UniqueName: \"kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.548277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72g86\" (UniqueName: \"kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.548789 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548768 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548929 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.548975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549338 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.549408 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfxw\" (UniqueName: \"kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rfz\" (UniqueName: \"kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549783 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550154 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.549987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550056 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.550673 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.550228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lm9v\" (UniqueName: \"kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.552162 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552145 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.552332 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552316 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:44:08.552639 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.552639 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552631 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:44:08.552769 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552683 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.552839 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552815 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\"" Nov 03 18:44:08.552890 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552827 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:44:08.552890 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552829 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.552957 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552889 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.552957 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552944 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:44:08.553020 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552979 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.553020 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.552982 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:44:08.553077 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.553060 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\"" Nov 03 18:44:08.553110 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.553089 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:08.553217 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.553179 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:44:08.553263 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.553240 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:44:08.553301 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.553286 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lqn4p" Nov 03 18:44:08.613345 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.613320 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:08.613345 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.613344 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:08.613514 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.613355 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:08.613514 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.613415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:09.113397582 +0000 UTC m=+3.126537629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:08.616065 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.616048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddbr\" (UniqueName: \"kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.651260 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.651319 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfxw\" (UniqueName: \"kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.651319 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.651402 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.651461 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.651440 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:08.651514 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651454 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.651608 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.651719 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.651719 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.651719 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.651685 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:09.151662082 +0000 UTC m=+3.164802100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:08.651859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.651859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.651859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rfz\" (UniqueName: \"kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.651859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651931 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651965 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651969 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651967 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652061 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652263 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.651832 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652359 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652358 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lm9v\" (UniqueName: \"kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.652459 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652459 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652640 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652672 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9cl\" (UniqueName: \"kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652682 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652802 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652821 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652843 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652847 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.653167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652964 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.652980 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.652994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653031 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:08.653043 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:09.153024649 +0000 UTC m=+3.166164666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq8w\" (UniqueName: \"kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653158 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.653998 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653423 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653463 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653464 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653509 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653564 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjrt\" (UniqueName: \"kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653644 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtlq\" (UniqueName: \"kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.654753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653852 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.653940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72g86\" (UniqueName: \"kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654706 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.655583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.654974 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655019 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655072 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655173 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655181 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.655792 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.656320 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.656234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.657035 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.656274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.657035 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.656731 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.657035 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.656999 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.720261 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.720229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rfz\" (UniqueName: \"kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:08.720907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.720878 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfxw\" (UniqueName: \"kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.720997 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.720975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9cl\" (UniqueName: \"kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.721043 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.721021 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lm9v\" (UniqueName: \"kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:44:08.721086 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.721032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjrt\" (UniqueName: \"kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.721086 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.721036 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq8w\" (UniqueName: \"kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.721602 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.721582 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtlq\" (UniqueName: \"kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.722041 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.722024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72g86\" (UniqueName: \"kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.747447 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.747416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:44:08.757884 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.757862 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40ce628_5272_4a70_a626_a48e7f83059b.slice/crio-6f4e86c59b1d92b64eb99204ed23d35bbe7f9d6cdddb297ea59e0342fdf159e5 WatchSource:0}: Error finding container 6f4e86c59b1d92b64eb99204ed23d35bbe7f9d6cdddb297ea59e0342fdf159e5: Status 404 returned error can't find the container with id 6f4e86c59b1d92b64eb99204ed23d35bbe7f9d6cdddb297ea59e0342fdf159e5 Nov 03 18:44:08.774776 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.774757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-222xl" Nov 03 18:44:08.777492 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.777473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:44:08.778808 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.778794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:08.785916 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.785898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:44:08.787127 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.787098 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fdd90b_d6c2_433a_a278_c8d3da866011.slice/crio-5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab WatchSource:0}: Error finding container 5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab: Status 404 returned error can't find the container with id 5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab Nov 03 18:44:08.787243 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.787140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:08.787847 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.787828 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc661c7b5_69d3_4001_8552_ac76facdb2d5.slice/crio-609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9 WatchSource:0}: Error finding container 609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9: Status 404 returned error can't find the container with id 609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9 Nov 03 18:44:08.790356 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.790315 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:08.792035 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.792008 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12643498_998e_4478_9045_6c384c87b101.slice/crio-989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874 WatchSource:0}: Error finding container 989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874: Status 404 returned error can't find the container with id 989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874 Nov 03 18:44:08.798163 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.798141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:44:08.798386 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.798368 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f6b64e_a198_460e_9c07_3eba0cc90e9a.slice/crio-6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72 WatchSource:0}: Error finding container 6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72: Status 404 returned error can't find the container with id 6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72 Nov 03 18:44:08.800031 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.800015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:44:08.800283 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.800262 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e2360c_6ea0_40aa_835d_dc764716cd98.slice/crio-0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00 WatchSource:0}: Error finding container 0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00: Status 404 returned error can't find the container with id 0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00 Nov 03 18:44:08.802442 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:08.802424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:44:08.808640 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.808622 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb78b5f_402b_40c9_9378_3eda2ddec2b9.slice/crio-eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb WatchSource:0}: Error finding container eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb: Status 404 returned error can't find the container with id eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb Nov 03 18:44:08.811641 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.811614 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9666f_95f8_4f02_b87f_1af832cce56f.slice/crio-2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b WatchSource:0}: Error finding container 2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b: Status 404 returned error can't find the container with id 2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b Nov 03 18:44:08.814635 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:08.814616 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99b660b_6975_4a0d_819a_1c9118904b50.slice/crio-33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5 WatchSource:0}: Error finding container 33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5: Status 404 returned error can't find the container with id 33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5 Nov 03 18:44:09.159391 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.159347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:09.159596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.159403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:09.159596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.159449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:09.159596 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159518 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159593 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159616 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159630 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159628 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159596 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:10.159572829 +0000 UTC m=+4.172712865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159691 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:10.159674116 +0000 UTC m=+4.172814141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:09.159772 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:09.159713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:10.159700797 +0000 UTC m=+4.172840815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:09.466942 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.466801 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-222xl" event={"ID":"e5fdd90b-d6c2-433a-a278-c8d3da866011","Type":"ContainerStarted","Data":"5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab"} Nov 03 18:44:09.468243 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.468214 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js4mm" event={"ID":"a40ce628-5272-4a70-a626-a48e7f83059b","Type":"ContainerStarted","Data":"6f4e86c59b1d92b64eb99204ed23d35bbe7f9d6cdddb297ea59e0342fdf159e5"} Nov 03 18:44:09.469660 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.469633 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerStarted","Data":"33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5"} Nov 03 18:44:09.470675 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.470657 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pxj4z" event={"ID":"b6d9666f-95f8-4f02-b87f-1af832cce56f","Type":"ContainerStarted","Data":"2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b"} Nov 03 18:44:09.471692 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.471674 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnctw" event={"ID":"36e2360c-6ea0-40aa-835d-dc764716cd98","Type":"ContainerStarted","Data":"0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00"} Nov 03 18:44:09.472826 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.472807 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9xv5" event={"ID":"e7f6b64e-a198-460e-9c07-3eba0cc90e9a","Type":"ContainerStarted","Data":"6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72"} Nov 03 18:44:09.473917 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.473894 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874"} Nov 03 18:44:09.475289 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.475256 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" event={"ID":"c661c7b5-69d3-4001-8552-ac76facdb2d5","Type":"ContainerStarted","Data":"609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9"} Nov 03 18:44:09.476926 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.476898 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvjcd" event={"ID":"8fb78b5f-402b-40c9-9378-3eda2ddec2b9","Type":"ContainerStarted","Data":"eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb"} Nov 03 18:44:09.556468 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.556430 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:08 +0000 UTC" deadline="2028-04-25 17:52:54.552701085 +0000 UTC" Nov 03 18:44:09.556468 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:09.556460 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12935h8m44.996244116s" Nov 03 18:44:10.166781 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.166740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:10.167020 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.166810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:10.167020 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.166856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:10.167020 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.166990 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:10.167150 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167055 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:12.167032551 +0000 UTC m=+6.180172571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:10.167150 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167147 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:10.167276 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167162 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:10.167276 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167195 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:10.167276 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:12.167224009 +0000 UTC m=+6.180364044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:10.167439 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167305 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:10.167439 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.167339 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:12.167328223 +0000 UTC m=+6.180468248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:10.457376 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.457296 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:10.457561 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.457438 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:10.457561 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.457535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:10.457688 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.457615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:10.457688 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.457685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:10.457803 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:10.457766 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:10.557159 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.557051 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:08 +0000 UTC" deadline="2028-07-11 23:25:47.408724718 +0000 UTC" Nov 03 18:44:10.557159 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:10.557113 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14788h41m36.851616085s" Nov 03 18:44:12.187571 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.187535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.187590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.187617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187727 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.187769562 +0000 UTC m=+10.200909579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187847 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187857 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187867 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187909 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.187883703 +0000 UTC m=+10.201023735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187952 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:12.188252 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.187976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.187968095 +0000 UTC m=+10.201108113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:12.459307 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.459230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:12.459471 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.459381 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:12.459471 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.459411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:12.459593 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.459516 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:12.459593 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:12.459550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:12.459741 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:12.459701 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:13.503797 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:13.503562 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="8ca229ee2eed349defc77fef1c3245df687446f7cbdcf3115399ce2a7f069702" exitCode=0 Nov 03 18:44:13.504333 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:13.503718 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"8ca229ee2eed349defc77fef1c3245df687446f7cbdcf3115399ce2a7f069702"} Nov 03 18:44:13.507345 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:13.507262 2580 generic.go:356] "Generic (PLEG): container finished" podID="c1266a5d4aae1fd29adb6e6003e98506" containerID="b13ab6f55abab0aa6c74c64e78bba44564226f615ede6a04ccec4ebea440531c" exitCode=0 Nov 03 18:44:13.507345 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:13.507299 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerDied","Data":"b13ab6f55abab0aa6c74c64e78bba44564226f615ede6a04ccec4ebea440531c"} Nov 03 18:44:14.457430 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.457394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:14.457607 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:14.457540 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:14.457671 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.457612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:14.457731 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:14.457691 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:14.460101 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.460071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:14.460248 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:14.460183 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:14.517310 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.516478 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerStarted","Data":"1e1133620545e4f9f62d41a4962bbf5251f680c8fd2647ef14430eb457d0d82b"} Nov 03 18:44:14.522185 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.521686 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"0683633c577f920552f48984254eada9","Type":"ContainerStarted","Data":"dce109b7fc0ecaabd57977e2e3f62bb5e338b82e64fb0b3340b8293d69f1760b"} Nov 03 18:44:14.644878 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.644346 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" podStartSLOduration=6.644326175 podStartE2EDuration="6.644326175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:14.643343776 +0000 UTC m=+8.656483819" watchObservedRunningTime="2026-11-03 18:44:14.644326175 +0000 UTC m=+8.657466218" Nov 03 18:44:14.771543 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:14.770655 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" podStartSLOduration=6.770634211 podStartE2EDuration="6.770634211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:14.770203857 +0000 UTC m=+8.783343898" watchObservedRunningTime="2026-11-03 18:44:14.770634211 +0000 UTC m=+8.783774254" Nov 03 18:44:15.529656 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:15.529618 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnctw" event={"ID":"36e2360c-6ea0-40aa-835d-dc764716cd98","Type":"ContainerStarted","Data":"9937f450949049934791469316bccf24ffd163de383d2f0186547f9f7f7f6d9c"} Nov 03 18:44:15.537986 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:15.537954 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9xv5" event={"ID":"e7f6b64e-a198-460e-9c07-3eba0cc90e9a","Type":"ContainerStarted","Data":"d38717b9cd96e46f10ac40c6c091575f68de50c8e35c4b5234784a8551d043df"} Nov 03 18:44:15.784323 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:15.784229 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xnctw" podStartSLOduration=3.184276972 podStartE2EDuration="8.78420752s" totalImagesPullingTime="5.599930548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:15.657657151 +0000 UTC m=+9.670797195" watchObservedRunningTime="2026-11-03 18:44:15.78420752 +0000 UTC m=+9.797347563" Nov 03 18:44:16.224071 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.224035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:16.224257 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.224095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:16.224257 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.224141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:16.224257 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224244 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:16.224426 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224293 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:24.224281232 +0000 UTC m=+18.237421248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:16.224595 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224575 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:16.224665 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224617 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:24.224607836 +0000 UTC m=+18.237747865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:16.224721 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224668 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:16.224721 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224677 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:16.224721 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224685 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:16.224721 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.224708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:24.22469936 +0000 UTC m=+18.237839390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:16.458348 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.458313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:16.458513 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.458450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:16.458915 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.458892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:16.459002 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.458930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:16.459039 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.459022 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:16.459148 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:16.459108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:16.487508 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.487440 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:16.488191 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.488170 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:16.542363 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.542324 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" event={"ID":"c661c7b5-69d3-4001-8552-ac76facdb2d5","Type":"ContainerStarted","Data":"4a18fee44c59145fbef327314b8a2ab03b55a5a53463bf8d7ad94c5ecf424041"} Nov 03 18:44:16.607262 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:16.607205 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z9xv5" podStartSLOduration=3.220923157 podStartE2EDuration="9.607185555s" totalImagesPullingTime="6.386262398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:15.785001955 +0000 UTC m=+9.798141999" watchObservedRunningTime="2026-11-03 18:44:16.607185555 +0000 UTC m=+10.620325594" Nov 03 18:44:17.544732 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:17.544701 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:18.233819 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.233792 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:44:18.419172 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.418400 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:44:18.233834683Z","UUID":"8d35686a-1df2-4848-8c23-7d175123dce2","Handler":null,"Name":"","Endpoint":""} Nov 03 18:44:18.420156 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.420140 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:44:18.420258 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.420166 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:44:18.457412 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.457368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:18.457551 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:18.457502 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:18.457613 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.457577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:18.457684 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:18.457660 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:18.457817 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.457801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:18.457933 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:18.457895 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:18.548740 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.548701 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pxj4z" event={"ID":"b6d9666f-95f8-4f02-b87f-1af832cce56f","Type":"ContainerStarted","Data":"cb77603e5bd94fe2b22de24b3bf250cacb32522a69694deca5bac2d1d355a1fe"} Nov 03 18:44:18.551176 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.551108 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" event={"ID":"c661c7b5-69d3-4001-8552-ac76facdb2d5","Type":"ContainerStarted","Data":"5a9aecb6a1eab31c8436839d5edeea536d7f9cacd29ad2e502b6c8c5e5e259d5"} Nov 03 18:44:18.553525 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.553498 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-222xl" event={"ID":"e5fdd90b-d6c2-433a-a278-c8d3da866011","Type":"ContainerStarted","Data":"82d2d46a3359b5e2ce154ed3594874bf0cde82c579118e7a34dbdcf25c4f4e67"} Nov 03 18:44:18.555803 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.555774 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="c9df72e7bfc01ee95ebde1c6cb66e645b817ca56a45ba786ee1fd5804f41d29a" exitCode=0 Nov 03 18:44:18.555904 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.555809 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"c9df72e7bfc01ee95ebde1c6cb66e645b817ca56a45ba786ee1fd5804f41d29a"} Nov 03 18:44:18.667276 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.667232 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pxj4z" podStartSLOduration=2.867182841 podStartE2EDuration="11.667218922s" totalImagesPullingTime="8.800036081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:18.666311415 +0000 UTC m=+12.679451453" watchObservedRunningTime="2026-11-03 18:44:18.667218922 +0000 UTC m=+12.680358959" Nov 03 18:44:18.910883 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:18.910784 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-222xl" podStartSLOduration=2.739838519 podStartE2EDuration="11.910764278s" totalImagesPullingTime="9.170925759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:18.909686889 +0000 UTC m=+12.922826929" watchObservedRunningTime="2026-11-03 18:44:18.910764278 +0000 UTC m=+12.923904317" Nov 03 18:44:19.560270 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:19.559765 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvjcd" event={"ID":"8fb78b5f-402b-40c9-9378-3eda2ddec2b9","Type":"ContainerStarted","Data":"47fb9d155ed819da997d608cef946dc8bad2115f4191729f397c789d87fe2a07"} Nov 03 18:44:19.677633 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:19.677401 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dvjcd" podStartSLOduration=3.540202903 podStartE2EDuration="12.67738533s" totalImagesPullingTime="9.137182427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:19.676953917 +0000 UTC m=+13.690093961" watchObservedRunningTime="2026-11-03 18:44:19.67738533 +0000 UTC m=+13.690525367" Nov 03 18:44:20.458211 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.458188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:20.458321 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:20.458298 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:20.458394 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.458380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:20.458539 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:20.458466 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:20.458539 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.458524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:20.458654 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:20.458595 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:20.563349 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.563315 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js4mm" event={"ID":"a40ce628-5272-4a70-a626-a48e7f83059b","Type":"ContainerStarted","Data":"33f10ee0430d7d1f191edcf7ad95fa87ca50f877772c67220b02e6c931dcfa15"} Nov 03 18:44:20.565603 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.565576 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="9dee30c23539089d1d6cf849af4a95ffe5932aac358bb6dd338503f0daac4dcd" exitCode=0 Nov 03 18:44:20.565716 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.565670 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"9dee30c23539089d1d6cf849af4a95ffe5932aac358bb6dd338503f0daac4dcd"} Nov 03 18:44:20.567698 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.567671 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" event={"ID":"c661c7b5-69d3-4001-8552-ac76facdb2d5","Type":"ContainerStarted","Data":"756a047ceb04218b190b778dc0dfb145b9ed4e859f018cb71cf69592e69c4977"} Nov 03 18:44:20.734500 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.734401 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-js4mm" podStartSLOduration=2.143599554 podStartE2EDuration="13.734388584s" totalImagesPullingTime="11.59078903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:20.733933743 +0000 UTC m=+14.747073781" watchObservedRunningTime="2026-11-03 18:44:20.734388584 +0000 UTC m=+14.747528651" Nov 03 18:44:20.854055 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:20.854005 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" podStartSLOduration=2.658511093 podStartE2EDuration="13.853991881s" totalImagesPullingTime="11.195480788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:20.853868269 +0000 UTC m=+14.867008308" watchObservedRunningTime="2026-11-03 18:44:20.853991881 +0000 UTC m=+14.867131919" Nov 03 18:44:22.456675 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:22.456645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:22.457581 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:22.456682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:22.457581 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:22.456755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:22.457581 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:22.456881 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:22.457581 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:22.457283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:22.457581 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:22.457371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:22.575098 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:22.575062 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="d11a7369fe2d71bd5086c7d798ff771a26de1761b0a8d7d69b1c71e7f137924b" exitCode=0 Nov 03 18:44:22.575277 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:22.575134 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"d11a7369fe2d71bd5086c7d798ff771a26de1761b0a8d7d69b1c71e7f137924b"} Nov 03 18:44:23.604778 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:23.604747 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:23.605213 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:23.604901 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:23.605716 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:23.605699 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:44:24.288740 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.288503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.288787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288689 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288840 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.288849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288855 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:24.288930 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288934 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:24.289256 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288949 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:40.288925879 +0000 UTC m=+34.302065922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:24.289256 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288975 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:40.288962234 +0000 UTC m=+34.302102263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:24.289256 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.288973 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:24.289256 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.289036 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:40.289019087 +0000 UTC m=+34.302159109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:24.456985 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.456955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:24.456985 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.456978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:24.457202 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.457070 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:24.457202 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:24.457100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:24.457202 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.457186 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:24.457333 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:24.457270 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:25.583911 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.583733 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="f91ccf5a49f300c171ac6cdd42d8b965cbdfb66120d67d969d0e67d9031fbb4d" exitCode=0 Nov 03 18:44:25.584419 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.583804 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"f91ccf5a49f300c171ac6cdd42d8b965cbdfb66120d67d969d0e67d9031fbb4d"} Nov 03 18:44:25.586913 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.586892 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"33680fc9d0a87419e4d00a09e109ba699a89b86331356ded4b701d12861b9496"} Nov 03 18:44:25.587010 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.586964 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"9396e31cc3dbc3bcf270cd0351cd703d13b2f5d72a100cee61fe4ca135402eda"} Nov 03 18:44:25.587072 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.587015 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"6b2f5d324004fd4cb6a587ce006aa0ea147a309fc36cd2563dfe5b25b8faccdc"} Nov 03 18:44:25.587072 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.587063 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"54f2f01b16ecb121defd74aeecaf6d9e44a91ee6d7b2df1442b62b9c8455af6a"} Nov 03 18:44:25.587186 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.587110 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"4f1a940026634ccbd06a054ae7306ba0513cdaee42d0ba29d8bf7c4e02f647a3"} Nov 03 18:44:25.587186 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:25.587174 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"008ec40d2e3b1e2d0b2a30983c127a953bf1940792870d87cd2ee7fc7c8c9e71"} Nov 03 18:44:26.006840 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.006775 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pxj4z_b6d9666f-95f8-4f02-b87f-1af832cce56f/dns-node-resolver/0.log" Nov 03 18:44:26.457989 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.457954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:26.458153 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:26.458069 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:26.458153 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.458131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:26.458252 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.458187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:26.458282 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:26.458259 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:26.458334 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:26.458311 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:26.591907 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.591875 2580 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="969190d7abc9667a5646a2c5040e80de24a95c453b6e152460cb74eaae209ae7" exitCode=0 Nov 03 18:44:26.592369 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.591924 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerDied","Data":"969190d7abc9667a5646a2c5040e80de24a95c453b6e152460cb74eaae209ae7"} Nov 03 18:44:26.778351 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:26.778285 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9xv5_e7f6b64e-a198-460e-9c07-3eba0cc90e9a/node-ca/0.log" Nov 03 18:44:27.596874 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:27.596839 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75kw7" event={"ID":"e99b660b-6975-4a0d-819a-1c9118904b50","Type":"ContainerStarted","Data":"b99cb4ec6c4585d4ecb89ea6afb7af251b02e2d518f3bb79d5899ff1c43771b5"} Nov 03 18:44:27.599982 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:27.599959 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"4da3c6a2660068755eb078d8e3f6866e15a34b339a219df544d40838ba17ad88"} Nov 03 18:44:27.721805 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:27.721764 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-75kw7" podStartSLOduration=6.804429922 podStartE2EDuration="20.721751753s" totalImagesPullingTime="13.917321831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:27.721097415 +0000 UTC m=+21.734237455" watchObservedRunningTime="2026-11-03 18:44:27.721751753 +0000 UTC m=+21.734891792" Nov 03 18:44:28.456556 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:28.456521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:28.456556 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:28.456544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:28.456752 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:28.456647 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:28.456752 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:28.456699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:28.456890 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:28.456832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:28.456947 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:28.456927 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:29.606899 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:29.606867 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" event={"ID":"12643498-998e-4478-9045-6c384c87b101","Type":"ContainerStarted","Data":"f00f0a1d4ac353925ecf83d5e2640fe599ce93cbec10a043d591ac2e93d73c2a"} Nov 03 18:44:29.607486 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:29.607161 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:29.607486 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:29.607225 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:29.622404 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:29.622386 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:29.892271 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:29.892173 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" podStartSLOduration=7.143231648 podStartE2EDuration="22.892159687s" totalImagesPullingTime="15.748928039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:29.840228435 +0000 UTC m=+23.853368473" watchObservedRunningTime="2026-11-03 18:44:29.892159687 +0000 UTC m=+23.905299750" Nov 03 18:44:30.456722 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:30.456690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:30.456722 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:30.456705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:30.457051 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:30.456801 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:30.457051 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:30.456850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:30.457051 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:30.456928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:30.457051 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:30.456995 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:30.609374 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:30.609339 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:30.623356 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:30.623334 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:44:32.457167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:32.457115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:32.457541 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:32.457177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:32.457541 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:32.457177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:32.457541 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:32.457257 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:32.457541 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:32.457341 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:32.457541 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:32.457401 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:34.457321 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:34.457291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:34.457321 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:34.457319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:34.457878 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:34.457405 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:34.457878 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:34.457450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:34.457878 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:34.457481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:34.457878 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:34.457558 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:36.457857 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:36.457822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:36.458414 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:36.457904 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:36.458414 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:36.457947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:36.458414 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:36.457986 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:36.458414 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:36.458016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:36.458414 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:36.458063 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:38.457157 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:38.457109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:38.457591 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:38.457110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:38.457591 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:38.457252 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:38.457591 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:38.457135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:38.457591 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:38.457303 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:38.457591 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:38.457389 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:40.304767 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.304735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.304777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.304813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304899 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304928 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304952 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304965 2580 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8q for pod openshift-network-diagnostics/network-check-target-dbs68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304964 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.304941 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret podName:8183d10b-0438-45e9-8e48-02c1cd3cc9f5 nodeName:}" failed. No retries permitted until 2026-11-03 18:45:12.304928114 +0000 UTC m=+66.318068131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret") pod "global-pull-secret-syncer-x7248" (UID: "8183d10b-0438-45e9-8e48-02c1cd3cc9f5") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.305029 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q podName:9609bd5a-37de-4bb8-921b-49c5885e4817 nodeName:}" failed. No retries permitted until 2026-11-03 18:45:12.305009893 +0000 UTC m=+66.318149910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8q" (UniqueName: "kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q") pod "network-check-target-dbs68" (UID: "9609bd5a-37de-4bb8-921b-49c5885e4817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:40.305238 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.305065 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs podName:2b2efaa8-4e8a-4f95-b766-6331405d9856 nodeName:}" failed. No retries permitted until 2026-11-03 18:45:12.305051581 +0000 UTC m=+66.318191608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs") pod "network-metrics-daemon-gvgvl" (UID: "2b2efaa8-4e8a-4f95-b766-6331405d9856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:40.457371 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.457336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:40.457371 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.457347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:40.457371 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:40.457370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:40.457586 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.457435 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:40.457586 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.457499 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:40.457586 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:40.457563 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:42.457203 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:42.457166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:42.457203 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:42.457194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:42.457701 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:42.457167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:42.457701 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:42.457282 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:42.457701 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:42.457344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:42.457701 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:42.457404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:44.456881 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:44.456846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:44.456881 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:44.456875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:44.457423 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:44.456902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:44.457423 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:44.457009 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:44.457423 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:44.457045 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:44.457423 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:44.457099 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:46.458847 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:46.458821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:46.459380 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:46.458930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:46.459380 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:46.458971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:46.459380 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:46.458959 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:46.459380 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:46.459038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:46.459380 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:46.459156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:47.685134 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.685077 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7248"] Nov 03 18:44:47.685542 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.685241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:47.685542 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:47.685340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:47.687564 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.687535 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbs68"] Nov 03 18:44:47.687704 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.687641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:47.687794 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:47.687756 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:47.688202 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.688187 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvgvl"] Nov 03 18:44:47.688278 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:47.688267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:47.688364 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:47.688349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:49.457438 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:49.457215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:49.457848 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:49.457253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:49.457848 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:49.457464 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:49.457848 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:49.457304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:49.457848 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:49.457568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:49.457848 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:49.457626 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:51.456808 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:51.456779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:51.457195 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:51.456779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:51.457195 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:51.456888 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:51.457195 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:51.456891 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7248" podUID="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" Nov 03 18:44:51.457195 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:51.456935 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbs68" podUID="9609bd5a-37de-4bb8-921b-49c5885e4817" Nov 03 18:44:51.457195 ip-10-0-133-236 kubenswrapper[2580]: E1103 18:44:51.457019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvgvl" podUID="2b2efaa8-4e8a-4f95-b766-6331405d9856" Nov 03 18:44:52.796398 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.796371 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeReady" Nov 03 18:44:52.796840 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.796490 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 18:44:52.915016 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.914947 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6lk6"] Nov 03 18:44:52.916824 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.916809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:52.966912 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.966875 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2c66j"] Nov 03 18:44:52.975141 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.975095 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:44:52.975290 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.975179 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:44:52.975290 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.975209 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\"" Nov 03 18:44:52.975393 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.974715 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:44:52.976845 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.976826 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hlbp5"] Nov 03 18:44:52.976940 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.976927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:52.978769 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.978751 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6lk6"] Nov 03 18:44:52.978859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.978777 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2c66j"] Nov 03 18:44:52.978859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.978797 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlbp5"] Nov 03 18:44:52.978859 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:52.978852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.073502 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073475 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:44:53.073706 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073679 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:44:53.073706 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073700 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:44:53.073871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073712 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:44:53.073871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073724 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:44:53.073871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073727 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\"" Nov 03 18:44:53.073871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073777 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\"" Nov 03 18:44:53.073871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.073850 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:44:53.099823 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.099802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.099902 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.099828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.099902 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.099852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.099902 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.099870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8gr\" (UniqueName: \"kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.100003 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.099987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.100033 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.100033 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.100158 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.100158 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjgf\" (UniqueName: \"kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.100223 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.100223 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.100204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tsj\" (UniqueName: \"kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.201374 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.201374 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.201588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjgf\" (UniqueName: \"kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.201588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.201588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tsj\" (UniqueName: \"kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.201743 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.201743 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.201743 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.201743 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8gr\" (UniqueName: \"kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.201926 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.201926 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.201816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.202075 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.202022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.202075 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.202055 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.202237 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.202214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.202296 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.202246 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.202296 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.202259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.205365 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.205343 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.205445 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.205369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.205445 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.205369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.264341 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.264311 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8gr\" (UniqueName: \"kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.264468 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.264390 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tsj\" (UniqueName: \"kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.264506 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.264489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjgf\" (UniqueName: \"kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.289857 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.289833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:44:53.293113 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.293088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:53.457457 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.457377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:44:53.457457 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.457404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:44:53.457671 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.457645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:44:53.512158 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512138 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:44:53.512158 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512143 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:44:53.512316 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512250 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:44:53.512316 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512298 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\"" Nov 03 18:44:53.512420 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512341 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\"" Nov 03 18:44:53.512420 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512366 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:44:53.512420 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.512404 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\"" Nov 03 18:44:53.527454 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.527438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:44:53.676599 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.676577 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2c66j"] Nov 03 18:44:53.726696 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.726675 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlbp5"] Nov 03 18:44:53.728371 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:53.728348 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a6de69_0836_498b_bd0e_704551c14e18.slice/crio-c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd WatchSource:0}: Error finding container c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd: Status 404 returned error can't find the container with id c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd Nov 03 18:44:53.891451 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:53.891419 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6lk6"] Nov 03 18:44:53.894151 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:53.894109 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008845af_8b96_4e32_acca_071973319800.slice/crio-9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63 WatchSource:0}: Error finding container 9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63: Status 404 returned error can't find the container with id 9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63 Nov 03 18:44:54.658886 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:54.658853 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6lk6" event={"ID":"008845af-8b96-4e32-acca-071973319800","Type":"ContainerStarted","Data":"9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63"} Nov 03 18:44:54.660185 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:54.660152 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlbp5" event={"ID":"e9a6de69-0836-498b-bd0e-704551c14e18","Type":"ContainerStarted","Data":"c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd"} Nov 03 18:44:54.661744 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:54.661719 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2c66j" event={"ID":"979d9146-3563-412a-8fe7-957dd49031aa","Type":"ContainerStarted","Data":"ca9e5e19d9188bb60f4c1dabceb2823ffe9a6d9e59d63b8566e8654cd4d097df"} Nov 03 18:44:54.661855 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:54.661764 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2c66j" event={"ID":"979d9146-3563-412a-8fe7-957dd49031aa","Type":"ContainerStarted","Data":"ac2bfd9e018f2ef681860ca91fe56c630141ce2795718eaaa983e364d5c52a10"} Nov 03 18:44:55.666720 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:55.666673 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2c66j" event={"ID":"979d9146-3563-412a-8fe7-957dd49031aa","Type":"ContainerStarted","Data":"d3b09356a1580625d311ae448f4cf67ce427da371d0469095ead4db54a305eec"} Nov 03 18:44:56.672218 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:56.672181 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlbp5" event={"ID":"e9a6de69-0836-498b-bd0e-704551c14e18","Type":"ContainerStarted","Data":"c0bf29c1812cd34a2549320ef493ea65226ff9806153fefdcd51d2761e47d32b"} Nov 03 18:44:56.672696 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:56.672244 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlbp5" event={"ID":"e9a6de69-0836-498b-bd0e-704551c14e18","Type":"ContainerStarted","Data":"55bd1896f8371dfa5f4c6718987b6d1c9d9abe23281c2c50e2e1816e2f16153f"} Nov 03 18:44:56.672696 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:56.672510 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hlbp5" Nov 03 18:44:56.793757 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:56.793696 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hlbp5" podStartSLOduration=2.766968383 podStartE2EDuration="4.793675924s" totalImagesPullingTime="2.026707541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:56.792909384 +0000 UTC m=+50.806049422" watchObservedRunningTime="2026-11-03 18:44:56.793675924 +0000 UTC m=+50.806815974" Nov 03 18:44:57.677272 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:57.677237 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6lk6" event={"ID":"008845af-8b96-4e32-acca-071973319800","Type":"ContainerStarted","Data":"32a6770ebafa18f977f63b9086f77edab6600efab4ce6b9570a50f223cea2112"} Nov 03 18:44:57.679445 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:57.679419 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2c66j" event={"ID":"979d9146-3563-412a-8fe7-957dd49031aa","Type":"ContainerStarted","Data":"ff4dd119c38770ebbac637cf745b02025745acf0194792be2f14f5b412fcc603"} Nov 03 18:44:57.797679 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:57.797625 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6lk6" podStartSLOduration=2.125009645 podStartE2EDuration="5.797612311s" totalImagesPullingTime="3.672602666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:57.797191172 +0000 UTC m=+51.810331210" watchObservedRunningTime="2026-11-03 18:44:57.797612311 +0000 UTC m=+51.810752348" Nov 03 18:44:57.917807 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:57.917754 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2c66j" podStartSLOduration=2.774905445 podStartE2EDuration="5.917741212s" totalImagesPullingTime="3.142835767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:57.91728733 +0000 UTC m=+51.930427379" watchObservedRunningTime="2026-11-03 18:44:57.917741212 +0000 UTC m=+51.930881250" Nov 03 18:44:58.460337 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.460307 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr"] Nov 03 18:44:58.462029 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.462015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.466449 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.466429 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4shpc"] Nov 03 18:44:58.468171 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.468156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.515543 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.515518 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:44:58.515853 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.515835 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 18:44:58.516107 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.516092 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dz7cd\"" Nov 03 18:44:58.541514 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7dr\" (UniqueName: \"kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541588 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541699 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541699 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541699 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541793 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541793 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541793 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxlz\" (UniqueName: \"kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.541988 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.541988 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.541964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.549298 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.549276 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:44:58.549380 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.549315 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:44:58.567386 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.567364 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 18:44:58.567486 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.567412 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:44:58.567938 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.567925 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:44:58.568006 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.567939 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:44:58.568058 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.568017 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltsf8\"" Nov 03 18:44:58.568283 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.568265 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr"] Nov 03 18:44:58.639806 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.639783 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:44:58.643022 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7dr\" (UniqueName: \"kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643084 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643084 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643177 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643177 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643177 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643151 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643177 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643294 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643294 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643294 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643294 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643247 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643294 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643453 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643453 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxlz\" (UniqueName: \"kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643453 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.643453 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643453 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643645 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643848 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643820 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.643975 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.643952 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.644157 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.644135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.644238 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.644205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.644305 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.644261 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.644366 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.644336 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.645871 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.645849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.645958 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.645855 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.645958 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.645933 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.646112 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.646094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.704872 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.704840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7dr\" (UniqueName: \"kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.705214 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.704993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxlz\" (UniqueName: \"kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.774255 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.774185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:44:58.780485 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:58.780465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:44:58.788452 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:58.788408 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32ce1b3_62ad_4273_8763_997294b83b96.slice/crio-a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf WatchSource:0}: Error finding container a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf: Status 404 returned error can't find the container with id a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf Nov 03 18:44:59.194555 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:59.194532 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr"] Nov 03 18:44:59.196733 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:44:59.196706 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9cc12e_64ec_4536_8e8d_02f28afcd385.slice/crio-228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff WatchSource:0}: Error finding container 228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff: Status 404 returned error can't find the container with id 228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff Nov 03 18:44:59.685063 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:59.685025 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" event={"ID":"ad9cc12e-64ec-4536-8e8d-02f28afcd385","Type":"ContainerStarted","Data":"228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff"} Nov 03 18:44:59.686181 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:44:59.686158 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4shpc" event={"ID":"b32ce1b3-62ad-4273-8763-997294b83b96","Type":"ContainerStarted","Data":"a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf"} Nov 03 18:45:00.691142 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:00.691060 2580 generic.go:356] "Generic (PLEG): container finished" podID="b32ce1b3-62ad-4273-8763-997294b83b96" containerID="7127d92423f3776f4b048d1156d8fee939c32a6d1df87be4091af472f883c722" exitCode=0 Nov 03 18:45:00.691142 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:00.691111 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4shpc" event={"ID":"b32ce1b3-62ad-4273-8763-997294b83b96","Type":"ContainerDied","Data":"7127d92423f3776f4b048d1156d8fee939c32a6d1df87be4091af472f883c722"} Nov 03 18:45:01.701169 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.701105 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4shpc" event={"ID":"b32ce1b3-62ad-4273-8763-997294b83b96","Type":"ContainerStarted","Data":"2096c5b911c2083a27c26808e76f390c774a536c52b7c98ecf95989f2a42b1f1"} Nov 03 18:45:01.701765 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.701195 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4shpc" event={"ID":"b32ce1b3-62ad-4273-8763-997294b83b96","Type":"ContainerStarted","Data":"e0c18634983b6201898d621a08cfca686518813375a0b85d5f24f7331a0dd8c3"} Nov 03 18:45:01.703965 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.703940 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" event={"ID":"ad9cc12e-64ec-4536-8e8d-02f28afcd385","Type":"ContainerStarted","Data":"60ee820e353b1772addd0b38a9ba7a85ce7c58d112420ee68d4006a9ef3d3474"} Nov 03 18:45:01.704094 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.703988 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" event={"ID":"ad9cc12e-64ec-4536-8e8d-02f28afcd385","Type":"ContainerStarted","Data":"ae61f7be22a93a83fae06099b0d314f45c196883ea2944a63c45764ecaffaf9c"} Nov 03 18:45:01.704094 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.704013 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" event={"ID":"ad9cc12e-64ec-4536-8e8d-02f28afcd385","Type":"ContainerStarted","Data":"fb18e755bd98748104edf0614adcfbeb3e8897ac4618be0f675149db85d345ce"} Nov 03 18:45:01.822657 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.822604 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4shpc" podStartSLOduration=2.199890111 podStartE2EDuration="3.822588731s" totalImagesPullingTime="1.54714095s" totalInitContainerRuntime="75.55767ms" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:01.821796796 +0000 UTC m=+55.834936834" watchObservedRunningTime="2026-11-03 18:45:01.822588731 +0000 UTC m=+55.835728766" Nov 03 18:45:01.944792 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:01.944728 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" podStartSLOduration=2.1442211 podStartE2EDuration="3.944713422s" totalImagesPullingTime="1.800492322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:01.943560509 +0000 UTC m=+55.956700546" watchObservedRunningTime="2026-11-03 18:45:01.944713422 +0000 UTC m=+55.957853459" Nov 03 18:45:02.623136 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:02.623070 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:03.417845 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.417812 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85778475ff-j2r6b"] Nov 03 18:45:03.419730 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.419715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.474003 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.473967 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zgffh\"" Nov 03 18:45:03.474003 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.473968 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 18:45:03.474003 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.473967 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2o4ukbnja9n82\"" Nov 03 18:45:03.476788 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.476768 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 18:45:03.477245 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.477231 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 18:45:03.477362 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.477316 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 18:45:03.478536 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.478514 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85778475ff-j2r6b"] Nov 03 18:45:03.483622 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.483706 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv76w\" (UniqueName: \"kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.483794 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.483852 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.483852 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.483924 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.483890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.484022 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.484008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.584893 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.584829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585074 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.584907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv76w\" (UniqueName: \"kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585074 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.584999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585074 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.585028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585074 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.585051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585321 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.585076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585321 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.585109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.585584 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.585537 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.586324 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.586301 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.586663 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.586641 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.587759 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.587737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.588014 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.587993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.588065 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.588049 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.647891 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.647864 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv76w\" (UniqueName: \"kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:03.729688 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:03.729657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:04.118583 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.118550 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c7dc9f994-29p67"] Nov 03 18:45:04.124487 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.124457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.150013 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.149977 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85778475ff-j2r6b"] Nov 03 18:45:04.171816 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:45:04.171785 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf815fdf2_7e68_4622_a30c_0e864a88b779.slice/crio-3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171 WatchSource:0}: Error finding container 3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171: Status 404 returned error can't find the container with id 3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171 Nov 03 18:45:04.182605 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.182583 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 18:45:04.182731 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.182708 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 18:45:04.182790 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.182709 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 18:45:04.182790 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.182763 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mrvdd\"" Nov 03 18:45:04.182863 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.182810 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 18:45:04.191487 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191577 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191577 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191645 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191681 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191681 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191776 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvp5\" (UniqueName: \"kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.191832 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.191815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.235041 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.235011 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 18:45:04.285923 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.285886 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 18:45:04.286994 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.286973 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c7dc9f994-29p67"] Nov 03 18:45:04.292297 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292369 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292369 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292490 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292543 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvp5\" (UniqueName: \"kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.292959 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.292934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.293282 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.293255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.293382 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.293329 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.294715 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.294670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.295428 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.295386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.295609 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.295509 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.295752 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.295732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.295848 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.295830 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.355862 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.355829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvp5\" (UniqueName: \"kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.437389 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.437302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:04.716167 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.716062 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" event={"ID":"f815fdf2-7e68-4622-a30c-0e864a88b779","Type":"ContainerStarted","Data":"3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171"} Nov 03 18:45:04.862538 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:04.862370 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c7dc9f994-29p67"] Nov 03 18:45:04.864958 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:45:04.864932 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9270f04e_1085_495a_9636_d4cba5958f5b.slice/crio-eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7 WatchSource:0}: Error finding container eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7: Status 404 returned error can't find the container with id eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7 Nov 03 18:45:05.720291 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:05.720255 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" event={"ID":"9270f04e-1085-495a-9636-d4cba5958f5b","Type":"ContainerStarted","Data":"eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7"} Nov 03 18:45:06.681785 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:06.681711 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:06.724815 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:06.724783 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" event={"ID":"f815fdf2-7e68-4622-a30c-0e864a88b779","Type":"ContainerStarted","Data":"97c14c373658008fae9dcd871f9eb02f4f72249e0c145205bfce4899e2c9c244"} Nov 03 18:45:06.923517 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:06.923468 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" podStartSLOduration=1.732529182 podStartE2EDuration="3.923454009s" totalImagesPullingTime="2.190924827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:06.922855237 +0000 UTC m=+60.935995272" watchObservedRunningTime="2026-11-03 18:45:06.923454009 +0000 UTC m=+60.936594113" Nov 03 18:45:09.735582 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:09.735540 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" event={"ID":"9270f04e-1085-495a-9636-d4cba5958f5b","Type":"ContainerStarted","Data":"6cea72b91398f24f1a87cac25b332373897b9c1cfc1c2dbae6cf996d038e5984"} Nov 03 18:45:10.740937 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:10.740898 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" event={"ID":"9270f04e-1085-495a-9636-d4cba5958f5b","Type":"ContainerStarted","Data":"0f0afb23cfcbc324d1c536843b00cfd40027ebc697dc71180c39cefaa9b128f3"} Nov 03 18:45:10.741300 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:10.740966 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" event={"ID":"9270f04e-1085-495a-9636-d4cba5958f5b","Type":"ContainerStarted","Data":"84aaed31f8c8d3d201e572f07edaf7b8bc3584d2b03e171d264fa8c8687909eb"} Nov 03 18:45:10.971971 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:10.971919 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" podStartSLOduration=1.600088425 podStartE2EDuration="6.971905744s" totalImagesPullingTime="5.371817319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:10.920003578 +0000 UTC m=+64.933143615" watchObservedRunningTime="2026-11-03 18:45:10.971905744 +0000 UTC m=+64.985045781" Nov 03 18:45:12.369829 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.369791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:12.370288 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.369859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:12.370288 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.369894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:12.423161 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.423107 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:45:12.423307 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.423214 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:45:12.432547 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.432516 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:12.474719 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.474698 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:45:12.482718 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.482692 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:12.483305 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.483289 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:45:12.493585 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.493563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:12.721450 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.721383 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\"" Nov 03 18:45:12.725768 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.725752 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\"" Nov 03 18:45:12.728894 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.728879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:12.731753 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.731741 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\"" Nov 03 18:45:12.733838 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.733820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:12.739596 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:12.739580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:13.103301 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.103275 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbs68"] Nov 03 18:45:13.145685 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:45:13.145657 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9609bd5a_37de_4bb8_921b_49c5885e4817.slice/crio-27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7 WatchSource:0}: Error finding container 27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7: Status 404 returned error can't find the container with id 27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7 Nov 03 18:45:13.154507 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.154441 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7248"] Nov 03 18:45:13.157692 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.157669 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvgvl"] Nov 03 18:45:13.160353 ip-10-0-133-236 kubenswrapper[2580]: W1103 18:45:13.160326 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2efaa8_4e8a_4f95_b766_6331405d9856.slice/crio-dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23 WatchSource:0}: Error finding container dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23: Status 404 returned error can't find the container with id dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23 Nov 03 18:45:13.755910 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.755871 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvgvl" event={"ID":"2b2efaa8-4e8a-4f95-b766-6331405d9856","Type":"ContainerStarted","Data":"dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23"} Nov 03 18:45:13.756985 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.756957 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbs68" event={"ID":"9609bd5a-37de-4bb8-921b-49c5885e4817","Type":"ContainerStarted","Data":"27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7"} Nov 03 18:45:13.757993 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:13.757969 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7248" event={"ID":"8183d10b-0438-45e9-8e48-02c1cd3cc9f5","Type":"ContainerStarted","Data":"c20ed09978b6b005d0dcbcbc556c00ccedd2042f0890937ef9dedb5625a50be6"} Nov 03 18:45:15.767611 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:15.767571 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvgvl" event={"ID":"2b2efaa8-4e8a-4f95-b766-6331405d9856","Type":"ContainerStarted","Data":"f0fcae6039647cd90f602ff3f008f3ea5d500721182bdfa23e377c70cab8015e"} Nov 03 18:45:15.768020 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:15.767635 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvgvl" event={"ID":"2b2efaa8-4e8a-4f95-b766-6331405d9856","Type":"ContainerStarted","Data":"1b40699db5bbbbef99071b32950716df9dfa753d86f88cc295cff3af8c2ef046"} Nov 03 18:45:15.887704 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:15.887647 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gvgvl" podStartSLOduration=67.187934078 podStartE2EDuration="1m8.887634162s" totalImagesPullingTime="1.699700084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:15.8874449 +0000 UTC m=+69.900584938" watchObservedRunningTime="2026-11-03 18:45:15.887634162 +0000 UTC m=+69.900774200" Nov 03 18:45:17.778520 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:17.778477 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbs68" event={"ID":"9609bd5a-37de-4bb8-921b-49c5885e4817","Type":"ContainerStarted","Data":"090ca630c0bc14923b7ea07df490ea83fc54a25e633cf72a9c2b9b52f0551fca"} Nov 03 18:45:17.778908 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:17.778603 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:17.896997 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:17.896945 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dbs68" podStartSLOduration=66.978675225 podStartE2EDuration="1m10.896932731s" totalImagesPullingTime="3.918257506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:17.896191444 +0000 UTC m=+71.909331485" watchObservedRunningTime="2026-11-03 18:45:17.896932731 +0000 UTC m=+71.910072794" Nov 03 18:45:18.517221 ip-10-0-133-236 kubenswrapper[2580]: I1103 18:45:18.517035 2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:45:18.517240 ip-10-0-133-236 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 18:45:18.537577 ip-10-0-133-236 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 18:45:18.537828 ip-10-0-133-236 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 18:45:18.537914 ip-10-0-133-236 systemd[1]: kubelet.service: Consumed 3.664s CPU time. Nov 03 18:45:18.559624 ip-10-0-133-236 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:45:18.619010 ip-10-0-133-236 kubenswrapper[5209]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:45:18.619010 ip-10-0-133-236 kubenswrapper[5209]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:45:18.619010 ip-10-0-133-236 kubenswrapper[5209]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:45:18.619010 ip-10-0-133-236 kubenswrapper[5209]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:45:18.621306 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621229 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.621306 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621304 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.621306 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621309 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621312 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621315 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621318 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621322 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621324 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621327 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621330 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621332 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621335 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621338 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621340 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621343 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621346 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621348 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621351 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621353 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621356 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621368 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621371 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.621420 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621374 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621376 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621379 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621382 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621384 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621387 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621389 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621392 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621395 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621397 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621400 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621402 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621405 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621408 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621411 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621413 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621416 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621424 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621427 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.621903 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621430 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621432 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621435 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621438 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621440 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621443 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621446 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621448 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621451 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621453 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621456 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621460 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621465 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621468 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621471 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621473 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621476 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621478 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621481 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621483 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.622386 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621486 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621489 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621492 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621494 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621497 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621500 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621503 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621505 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621508 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621511 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621513 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621521 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621523 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621526 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621529 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621531 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621534 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621536 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621539 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621541 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.622871 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621544 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621546 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621549 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621551 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621554 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621557 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621559 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621562 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621564 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621567 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621569 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621572 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621575 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621577 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621581 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621585 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621590 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621594 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621596 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.623366 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621599 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621601 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621604 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621606 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621609 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.621613 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622372 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622380 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622383 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622386 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622389 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622392 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622395 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622398 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622401 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622403 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622406 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622410 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622413 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622416 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.623839 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622418 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622421 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622424 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622426 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622429 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622432 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622434 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622438 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622441 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622443 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622446 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622448 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622451 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622454 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622456 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622459 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622461 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622464 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622467 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622470 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.624341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622472 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622475 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622478 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622480 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622483 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622486 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622489 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622491 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622494 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622497 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622499 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622502 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622504 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622507 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622509 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622512 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622514 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622517 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622519 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622523 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.624827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622525 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622528 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622530 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622533 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622535 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622538 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622541 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622543 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622546 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622548 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622551 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622554 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622558 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622561 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622564 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622567 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622570 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622572 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622575 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.625340 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622578 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622581 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622583 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622586 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622588 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622591 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622593 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622596 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622599 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622602 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622604 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622607 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622609 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622613 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622617 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622620 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622623 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622626 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622629 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622631 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.625827 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622634 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622636 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622643 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622645 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622648 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622651 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622654 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622656 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622659 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622662 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622664 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622667 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.622670 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623151 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623158 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623160 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623163 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623166 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623169 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623171 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.626339 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623174 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623177 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623180 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623183 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623186 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623189 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623192 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623194 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623197 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623199 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623202 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623204 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623207 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623209 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623214 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623218 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623222 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623226 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623229 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.626824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623231 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623234 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623236 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623239 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623242 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623245 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623249 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623253 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623256 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623259 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623261 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623264 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623266 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623269 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623272 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623275 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623277 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623280 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623282 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623285 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.627293 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623288 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623290 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623293 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623295 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623298 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623300 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623303 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623305 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623308 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623311 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623313 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623316 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623322 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623325 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623327 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623330 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623332 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623335 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623337 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.627769 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623340 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623343 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623345 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623348 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623350 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623353 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623355 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623358 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623361 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623363 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623366 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623368 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623371 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623374 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623377 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623379 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623382 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623384 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623387 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.628243 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623390 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623392 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623394 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623397 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623399 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623402 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623406 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623409 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623412 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623414 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623417 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623419 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623422 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623426 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623428 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623431 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623433 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623436 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623438 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623441 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.628711 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623444 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.623447 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623536 5209 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623553 5209 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623561 5209 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623568 5209 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623573 5209 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623576 5209 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623580 5209 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623585 5209 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623588 5209 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623591 5209 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623594 5209 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623597 5209 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623601 5209 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623604 5209 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623607 5209 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623610 5209 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623615 5209 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623618 5209 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623623 5209 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623626 5209 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623629 5209 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623632 5209 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:45:18.629436 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623636 5209 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623639 5209 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623643 5209 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623646 5209 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623649 5209 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623652 5209 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623656 5209 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623659 5209 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623662 5209 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623665 5209 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623669 5209 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623672 5209 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623675 5209 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623678 5209 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623682 5209 flags.go:64] FLAG: --enable-server="true" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623690 5209 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623694 5209 flags.go:64] FLAG: --event-burst="100" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623697 5209 flags.go:64] FLAG: --event-qps="50" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623701 5209 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623704 5209 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623707 5209 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623711 5209 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623714 5209 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623717 5209 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623720 5209 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:45:18.630223 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623723 5209 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623728 5209 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623731 5209 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623735 5209 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623738 5209 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623741 5209 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623744 5209 flags.go:64] FLAG: --feature-gates="" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623748 5209 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623751 5209 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623756 5209 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623759 5209 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623762 5209 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623765 5209 flags.go:64] FLAG: --help="false" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623768 5209 flags.go:64] FLAG: --hostname-override="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623771 5209 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623774 5209 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623777 5209 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623781 5209 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623784 5209 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623787 5209 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623790 5209 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623793 5209 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623796 5209 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:45:18.630849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623799 5209 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623803 5209 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623805 5209 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623809 5209 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623811 5209 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623814 5209 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623817 5209 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623820 5209 flags.go:64] FLAG: --lock-file="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623823 5209 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623826 5209 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623831 5209 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623837 5209 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623840 5209 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623844 5209 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623847 5209 flags.go:64] FLAG: --logging-format="text" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623850 5209 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623853 5209 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623856 5209 flags.go:64] FLAG: --manifest-url="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623861 5209 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623866 5209 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623869 5209 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623873 5209 flags.go:64] FLAG: --max-pods="110" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623876 5209 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623879 5209 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623882 5209 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:45:18.631421 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623885 5209 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623888 5209 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623891 5209 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623894 5209 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623900 5209 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623904 5209 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623907 5209 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623910 5209 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623913 5209 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623916 5209 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623919 5209 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623922 5209 flags.go:64] FLAG: --port="10250" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623925 5209 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623928 5209 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0907cfa58e4a560e9" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623931 5209 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623934 5209 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623937 5209 flags.go:64] FLAG: --register-node="true" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623942 5209 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623946 5209 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623949 5209 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623952 5209 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623956 5209 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623959 5209 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623963 5209 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623966 5209 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623970 5209 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:45:18.632063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623973 5209 flags.go:64] FLAG: --runonce="false" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623976 5209 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623980 5209 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623983 5209 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623986 5209 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623989 5209 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623992 5209 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623995 5209 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.623998 5209 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624001 5209 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624003 5209 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624006 5209 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624009 5209 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624012 5209 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624015 5209 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624018 5209 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624024 5209 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624026 5209 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624029 5209 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624034 5209 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624036 5209 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624039 5209 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624042 5209 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624047 5209 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624050 5209 flags.go:64] FLAG: --v="2" Nov 03 18:45:18.632735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624054 5209 flags.go:64] FLAG: --version="false" Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624058 5209 flags.go:64] FLAG: --vmodule="" Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624063 5209 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624068 5209 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624215 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624220 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624225 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624227 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624230 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624233 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624236 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624239 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624242 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624244 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624247 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624250 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624252 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624255 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624257 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624260 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.633391 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624263 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624265 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624268 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624270 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624273 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624275 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624278 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624281 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624283 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624286 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624290 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624292 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624295 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624298 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624300 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624303 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624306 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624309 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624315 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.633891 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624319 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624322 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624324 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624327 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624329 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624332 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624335 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624337 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624340 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624342 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624347 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624351 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624354 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624356 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624359 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624362 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624364 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624366 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624369 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.634502 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624372 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624374 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624377 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624380 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624383 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624386 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624389 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624391 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624394 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624397 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624399 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624402 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624406 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624408 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624411 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624414 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624416 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624419 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624421 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.635034 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624424 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624427 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624429 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624432 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624434 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624437 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624439 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624442 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624444 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624447 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624450 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624452 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624455 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624457 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624460 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624462 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624465 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624469 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624471 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624474 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.635803 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624476 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624479 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624482 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624484 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624487 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624491 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624493 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624496 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624499 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624501 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624504 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624507 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624509 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.624534 5209 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624569 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624573 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.636767 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624576 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624578 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624581 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624584 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624586 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624589 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624591 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624594 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624597 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624599 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624603 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624605 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624607 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624610 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624614 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624617 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624620 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624622 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624625 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.637321 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624627 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624630 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624634 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624637 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624640 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624642 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624645 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624648 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624650 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624653 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624655 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624658 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624661 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624663 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624666 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624668 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624671 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624673 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624677 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.637905 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624681 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624684 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624687 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624689 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624692 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624694 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624697 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624700 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624703 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624706 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624711 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624715 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624718 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624720 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624723 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624727 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624730 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624733 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624736 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.638398 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624738 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624741 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624744 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624746 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624749 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624752 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624754 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624756 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624759 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624762 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624764 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624767 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624770 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624772 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624775 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624777 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624780 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624782 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624792 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.638858 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624795 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624798 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624802 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624805 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624807 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624809 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624812 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624815 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624818 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624821 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624824 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624826 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624830 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624833 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624835 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624840 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624845 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624849 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624853 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624855 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.639324 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624858 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624861 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624863 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624866 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624868 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624871 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624873 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.624876 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.628330 5209 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.628344 5209 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.628362 5209 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.628368 5209 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628424 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628430 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628433 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628436 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628439 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628442 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628445 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.639831 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628448 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628451 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628454 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628457 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628460 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628463 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628466 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628469 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628471 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628474 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628477 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628479 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628482 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628484 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628487 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628490 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628493 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628496 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628499 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628502 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.640319 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628505 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628508 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628511 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628513 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628516 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628518 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628521 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628524 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628526 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628529 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628531 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628534 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628537 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628539 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628542 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628545 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628548 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628550 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628553 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628556 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.640814 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628558 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628561 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628564 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628566 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628569 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628572 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628574 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628577 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628579 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628582 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628585 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628588 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628591 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628595 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628598 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628601 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628604 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628606 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628609 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628612 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.641316 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628614 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628617 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628619 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628623 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628627 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628631 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628634 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628637 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628639 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628642 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628644 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628647 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628650 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628652 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628655 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628658 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628660 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628663 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628665 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.641783 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628668 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628670 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628673 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628676 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628679 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628681 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628684 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628686 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628689 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628691 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628693 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628696 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628699 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628701 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628704 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628706 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628709 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628711 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628714 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.642258 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.628717 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.628743 5209 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629204 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629408 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629413 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629418 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629423 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629428 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629432 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629438 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629442 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629447 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629451 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629462 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629466 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629471 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.642716 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629476 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629480 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629484 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629489 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629494 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629498 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629503 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629508 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629513 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629517 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629528 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629532 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629537 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629541 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629546 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629550 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629555 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629561 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629567 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.643099 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629573 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629578 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629582 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629592 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629597 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629601 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629606 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629610 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629614 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629619 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629623 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629628 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629633 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629637 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629642 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629646 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629655 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629659 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629665 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.643597 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629669 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629674 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629679 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629683 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629687 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629692 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629696 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629700 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629705 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629712 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629723 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629728 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629733 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629737 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629742 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629746 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629751 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629756 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629760 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629764 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.644044 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629774 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629806 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629828 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629862 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629976 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629985 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629992 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.629997 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630001 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630005 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630009 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630014 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630018 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630022 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630026 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630030 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630034 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630038 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630042 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630047 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.644533 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630051 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630055 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630060 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630064 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630069 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630073 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630078 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630082 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630086 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630090 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630095 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630099 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630104 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630108 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.630171 5209 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:45:18.645006 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630226 5209 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630234 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630238 5209 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630241 5209 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630244 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630247 5209 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630249 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630252 5209 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630254 5209 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630257 5209 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630260 5209 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630262 5209 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630265 5209 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630267 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630270 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630273 5209 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630275 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630278 5209 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630285 5209 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630287 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:45:18.645419 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630290 5209 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630293 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630295 5209 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630298 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630303 5209 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630306 5209 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630310 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630313 5209 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630316 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630318 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630321 5209 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630323 5209 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630326 5209 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630328 5209 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630331 5209 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630334 5209 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630336 5209 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630339 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630341 5209 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:45:18.645893 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630344 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630346 5209 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630349 5209 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630352 5209 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630354 5209 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630357 5209 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630360 5209 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630362 5209 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630365 5209 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630368 5209 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630370 5209 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630374 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630377 5209 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630379 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630382 5209 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630386 5209 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630390 5209 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630393 5209 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630396 5209 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630400 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:45:18.646370 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630403 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630406 5209 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630409 5209 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630412 5209 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630415 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630418 5209 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630420 5209 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630423 5209 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630425 5209 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630428 5209 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630431 5209 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630433 5209 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630436 5209 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630438 5209 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630441 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630444 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630447 5209 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630449 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630452 5209 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630454 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:45:18.646845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630457 5209 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630460 5209 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630462 5209 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630466 5209 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630468 5209 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630471 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630474 5209 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630476 5209 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630479 5209 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630481 5209 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630484 5209 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630487 5209 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630490 5209 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630492 5209 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630495 5209 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630497 5209 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630500 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630503 5209 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630505 5209 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:45:18.647341 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630508 5209 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630510 5209 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630513 5209 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630516 5209 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630518 5209 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630521 5209 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630523 5209 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:18.630526 5209 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.630577 5209 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631505 5209 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631569 5209 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631750 5209 server.go:1014] "Starting client certificate rotation" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631891 5209 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631932 5209 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:39:07 +0000 UTC" deadline="2028-05-24 03:14:00.67499321 +0000 UTC" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.631957 5209 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13616h28m42.043038179s" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.632269 5209 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:45:18.647782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.633598 5209 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:45:18.648397 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.634679 5209 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:45:18.648397 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.636820 5209 log.go:25] "Validated CRI v1 image API" Nov 03 18:45:18.648397 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.637563 5209 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:45:18.648397 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.639275 5209 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b011e213-6a53-4430-97c9-307d3a78024d:/dev/nvme0n1p4 d7b0a5ae-75d1-4816-9eb1-c0df023beac6:/dev/nvme0n1p3] Nov 03 18:45:18.648520 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.639296 5209 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac2bfd9e018f2ef681860ca91fe56c630141ce2795718eaaa983e364d5c52a10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac2bfd9e018f2ef681860ca91fe56c630141ce2795718eaaa983e364d5c52a10/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c20ed09978b6b005d0dcbcbc556c00ccedd2042f0890937ef9dedb5625a50be6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c20ed09978b6b005d0dcbcbc556c00ccedd2042f0890937ef9dedb5625a50be6/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~projected/kube-api-access-dmjgf:{mountpoint:/var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~projected/kube-api-access-dmjgf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~projected/kube-api-access-hn9cl:{mountpoint:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~projected/kube-api-access-hn9cl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~projected/kube-api-access-g8rfz:{mountpoint:/var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~projected/kube-api-access-g8rfz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~secret/metrics-certs major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36e2360c-6ea0-40aa-835d-dc764716cd98/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/36e2360c-6ea0-40aa-835d-dc764716cd98/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8183d10b-0438-45e9-8e48-02c1cd3cc9f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8183d10b-0438-45e9-8e48-02c1cd3cc9f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fb78b5f-402b-40c9-9378-3eda2ddec2b9/volumes/kubernetes.io~projected/kube-api-access-gvtlq:{mountpoint:/var/lib/kubelet/pods/8fb78b5f-402b-40c9-9378-3eda2ddec2b9/volumes/kubernetes.io~projected/kube-api-access-gvtlq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~projected/kube-api-access-fcvp5:{mountpoint:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~projected/kube-api-access-fcvp5 major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9609bd5a-37de-4bb8-921b-49c5885e4817/volumes/kubernetes.io~projected/kube-api-access-h4j8q:{mountpoint:/var/lib/kubelet/pods/9609bd5a-37de-4bb8-921b-49c5885e4817/volumes/kubernetes.io~projected/kube-api-access-h4j8q major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~projected/kube-api-access-f9tsj:{mountpoint:/var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~projected/kube-api-access-f9tsj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~projected/kube-api-access-bddbr:{mountpoint:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~projected/kube-api-access-bddbr major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~projected/kube-api-access-kn7dr:{mountpoint:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~projected/kube-api-access-kn7dr major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~projected/kube-api-access-tjxlz:{mountpoint:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~projected/kube-api-access-tjxlz major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d9666f-95f8-4f02-b87f-1af832cce56f/volumes/kubernetes.io~projected/kube-api-access-72g86:{mountpoint:/var/lib/kubelet/pods/b6d9666f-95f8-4f02-b87f-1af832cce56f/volumes/kubernetes.io~projected/kube-api-access-72g86 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c661c7b5-69d3-4001-8552-ac76facdb2d5/volumes/kubernetes.io~projected/kube-api-access-8pq8w:{mountpoint:/var/lib/kubelet/pods/c661c7b5-69d3-4001-8552-ac76facdb2d5/volumes/kubernetes.io~projected/kube-api-access-8pq8w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5fdd90b-d6c2-433a-a278-c8d3da866011/volumes/kubernetes.io~projected/kube-api-access-9lm9v:{mountpoint:/var/lib/kubelet/pods/e5fdd90b-d6c2-433a-a278-c8d3da866011/volumes/kubernetes.io~projected/kube-api-access-9lm9v major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f6b64e-a198-460e-9c07-3eba0cc90e9a/volumes/kubernetes.io~projected/kube-api-access-cgjrt:{mountpoint:/var/lib/kubelet/pods/e7f6b64e-a198-460e-9c07-3eba0cc90e9a/volumes/kubernetes.io~projected/kube-api-access-cgjrt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e99b660b-6975-4a0d-819a-1c9118904b50/volumes/kubernetes.io~projected/kube-api-access-bhfxw:{mountpoint:/var/lib/kubelet/pods/e99b660b-6975-4a0d-819a-1c9118904b50/volumes/kubernetes.io~projected/kube-api-access-bhfxw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~projected/kube-api-access-pn8gr:{mountpoint:/var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~projected/kube-api-access-pn8gr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~projected/kube-api-access-wv76w:{mountpoint:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~projected/kube-api-access-wv76w major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:287 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/52186815c4148f4f022a2d057c0bcfb4adc7d7fbbe6856bb8d01d2e1fa11b6c9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8422cea2c98ea23c3a22f34bc51518a19fe15369bc5a94023f21f16572326f6b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/591701601b74f01e2a0b3d4622704730f5dc2943c070cc983c5406d879c18f34/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7a5e22ccc8240a51441a4c999b9c45f20638ecfc93a8f2b2fe8c35ddcca91dfd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e77124e238204c0b3594e9a6bcc879e79c16ebea4889c25cc1d20e4181287e17/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1831f5a2ddc817644be7736838f13a34eaa453dbd0d6ed3cfd3a11164a375d5b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3d68e8cf278876167a877b3a1b17f51b37de9e7210399deeebf349c54b46b23a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d88e7de9dbb7e0989ff51670f2a5224a31d1927eb58c574669a2137f06b00b52/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4a6b84338f1b3ba31c95372b4264b708608c7cb4ede26e1855f36e4cc0ebf005/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fb08a37f1720b91143ea6a5d5f0aa6b332058f4729d262dec7d0578db1105018/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/fabd2228d6c36c8783d9e8a32661a6f07c58924314c26f52fb831a441d6f5612/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/492144aec31b30b4c856349962c4592b613209b3aa3984af2d56b5c184208d77/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/6aec48b7f9276dfb70661906e84e5ae5d571868ffc826124449d96029046e099/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/3a7b01293f9a69e761ed0c69d8c480b43fbfa6ae0e81041c119706075f61bfb6/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/615383acc4cad72573863330e6aeee325c014203be10aedbf27156aa07604916/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bc911ae817dbf4ea65370bb1dd54f9ff7847a2d30647d647db80c9c6471b1183/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/94ef3665c5614765b1c542fba222add802d1f7ae3bcd0d0320c26c373ac44f8a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d348a6a73297fadc19da00dfbac9b273a5d564f86ec45cf0590f0ed4ecbb761a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ad874f45327e276365ea90f324b35c8f4fbf774c04a65ccd2a6f34253cf5f03e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/71775356b95193919809c63b18a9831777f0dcc4a67394b1dfe85557c81e7073/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8e62af378f3512845146aabfc6ef2755f1071aab208e3d5794cc38a23d6ee924/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5fc37c29f2093cef5a5dde5d16cab4ff942c16ac145a69ee3895ee8904553aef/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/72fcd4b8019e83c3aaae5df36fb378860f865df188e9564cde24e48c39c08488/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c567196ec54dd85976463216fee880972a184d9317744a0f876a28525ab9d44f/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/2d5f85beefd2af0276cfbb17ec7cd1c2ff334377094dca87cbffef09193994dc/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/84dd584804f0993a27633dde1ae85bfd47e1195bb3206cb9b204290362b12a69/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/9b2eca1b96ac720b80a78a67e0c30ae095c9225d74dad11747b818791532b8e8/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d8360fab7e5cd251fb1d43df9e59ee8082afb4b8b3569756b4aeb5ceb7ece673/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/fae5f1ad4f2e3f76325034f270aa5eb0788912c6f2f20ecc761e2c8c2b0bf4d7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/6561a2c6c2e5c45cfbb68d0a5c7cca87bc0e4e185d9067cc570ecce7861347f2/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/53921a17c1638def6dba5cafae05b9d4f7d00fec0d6d50626d57e0fdc0ca4ef9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/38d1842c40b62b1b9a9e29e78dd4edc4d16fca434942e16ef24fe8439c153666/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/10ec3b70637090279cc57d20e773e13be1c30b25b34e9866bb47361336ed0d65/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/004232210f01ce8605c0401d639c1b22c6747c68a612a51c4abd86a75175526c/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/a997ddebb4a23015ac67571bbf2d0f647fa631be77918a3899ccdcd519170588/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/d88660d974b9074b03de2d94d986700c1ba41e67eaa83c0a4ae146071c050661/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/4c1e8b76f65b88dad0e2272f99fc40e33f50f6860f2179b0fac5b1654ae2ee6e/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/0b4e21a401dcceca36ea0c7d85ee9393c114425170dcd37ab464756e58d12ad0/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/5b8043fb4e4d60b0c7ab936762ba329e793343072e466f022661fd78790be693/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/a9e99bc04bb3d108062c07e5505a80fc7ce311cabb4a6dcd8df0353eb4f49381/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/468bcf8fa26c9860fc3e0fd116c0e678b250aaa85ca071fa9e29c3db69d85984/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/3a770771c0578f85ac436f2fcda536edccb3b265924721980840ab1be0bf5b95/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/93998510084af93328622fbf907d9e0e7466b37c786c7e466deb17603a8b3490/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/7c4de933150578916e68706ffdd56f91d08b0e074f2d6d6541e6ddaffff649dc/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/6365756fbe33d07328726df6b3d6f01616e31a428faff4da9de80e7c4b1589eb/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/46e2c737832ec072ed59284579b559fcdba6fc4389ee787f17cbab74d182313c/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/575ab3a55af14deedb0d05eb59b093032b93d95b373c140ba2c5a86495328798/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f5027c4d37b60900113513ff2d75fb756841b5bc34d16b20135ac53be10ef5eb/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/8bc7d32eaf8a12856e44e49d7a0874cb46bb17539720a0a0d46492fdd9f73819/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/b9cfbd312c24f9dbb9baead8074777eb03a23ef203a8cf39b9719a074a9de41a/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/90e9edd790345d975ec2d808ad73ad58a7269d64283815be9c57ce010c481ea1/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c8897e2a25686103444fc37a2b140fca940c8d891941c6ce75a726e5f2e8a20f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/1ac589aa9e82b761a2540ba4998fd3c71ceb99b693108485614e8df12d91f93e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5a1572d813bb9f5289d78c40d34d13fdc7038b837c3719c3f65c98ecce9ca425/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0de36a3a4ff9e84ab8ff43ea2f278d463569bad69dd1b5c9f4d6d433fa06a273/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a7032f585b7e8d3e10219d1e6e6e5e72d1c45f4c8eefb37f4a81b34001e9ea8b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d0f9bca76921184ae2876e0f9a07ca1cadcb0a7527066fcf7e37c04f18dbfe31/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a83430b65da5651988e7597dd6fc7de9ba1cc8aea855f60205d4709f4d08ddfa/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/153bf05462f320183644d5a59d1b491dbf8e959af8ed261511584f8aadf6b8d8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5ca79fa7107f73477acbea26bc412e06a8478fc9eb294739f98b26cfd0d22217/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.646637 5209 manager.go:217] Machine: {Timestamp:2026-11-03 18:45:18.646190694 +0000 UTC m=+0.057290808 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104776 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b696ce05d015ddc6f0c76e4e52a8e SystemUUID:ec2b696c-e05d-015d-dc6f-0c76e4e52a8e BootID:aa209c8a-993a-4ffe-8ffe-31c1566c39f2 Filesystems:[{Device:/run/containers/storage/overlay-containers/5ec282fc68c8813c718b823a4e1821ffbd92c4660cfa9a68d59071a09b020dab/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33cf8c36137b308fe41d02201b4705e5065cf01c128988bd7d88c7f7ca0040a5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~projected/kube-api-access-fcvp5 DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fb78b5f-402b-40c9-9378-3eda2ddec2b9/volumes/kubernetes.io~projected/kube-api-access-gvtlq DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c20ed09978b6b005d0dcbcbc556c00ccedd2042f0890937ef9dedb5625a50be6/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36e2360c-6ea0-40aa-835d-dc764716cd98/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5210266ed98c1d8e69deabd85e441fff23988db96431629bed6a81342c45ecf/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~projected/kube-api-access-dmjgf DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~projected/kube-api-access-tjxlz DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9609bd5a-37de-4bb8-921b-49c5885e4817/volumes/kubernetes.io~projected/kube-api-access-h4j8q DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b1fe65843ab833e77095532f3318964106e2506013345e45b0239679bf0cd00/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~projected/kube-api-access-f9tsj DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac2bfd9e018f2ef681860ca91fe56c630141ce2795718eaaa983e364d5c52a10/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/609a2a9fac2d3aa284153253b31e655643c87b41a6637e2965a02bada1f776c9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e94be77f42b5daad1f1149bc1447482ff55c508382078cbdcd0a258a9ca5a9b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/228f656ade4f3cba5c518a72bc0696ce034bef3d0e57c25615297b5311c2e2ff/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c11ab5db0e79af4fbf79fe397ffcb4760847e55544432615ab04241c404421fd/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d01e25af4198216e20011b355675c492458da91d3013d0f54e34c3c80ae5171/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27cd4538748a2a0ad726075758bf796bdb2c89be87e21ffa787f5041214679e7/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~projected/kube-api-access-kn7dr DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd7b7cedbcf465e6ad9e811adb49a6823fa2fa5d4875a65045e57a30aa6d8e23/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/008845af-8b96-4e32-acca-071973319800/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/979d9146-3563-412a-8fe7-957dd49031aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f6b64e-a198-460e-9c07-3eba0cc90e9a/volumes/kubernetes.io~projected/kube-api-access-cgjrt DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d9666f-95f8-4f02-b87f-1af832cce56f/volumes/kubernetes.io~projected/kube-api-access-72g86 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eab31272aca922f8142cbe7b05401fafe35e7e5baa5fbbb831d85e05367653fb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a40ce628-5272-4a70-a626-a48e7f83059b/volumes/kubernetes.io~projected/kube-api-access-bddbr DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb5f378e4ffec66e486c90c37342f6d2b4c2093dac36bc2e9e8541a6c575b6c7/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b32ce1b3-62ad-4273-8763-997294b83b96/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8183d10b-0438-45e9-8e48-02c1cd3cc9f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a6de69-0836-498b-bd0e-704551c14e18/volumes/kubernetes.io~projected/kube-api-access-pn8gr DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12643498-998e-4478-9045-6c384c87b101/volumes/kubernetes.io~projected/kube-api-access-hn9cl DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f9d72cd5249e45a3c1f0bbea4cc1a8de2c4b485cc65965ea5719c6cea8cdd63/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c661c7b5-69d3-4001-8552-ac76facdb2d5/volumes/kubernetes.io~projected/kube-api-access-8pq8w DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9cc12e-64ec-4536-8e8d-02f28afcd385/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5fdd90b-d6c2-433a-a278-c8d3da866011/volumes/kubernetes.io~projected/kube-api-access-9lm9v DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/989a5a05829341aecdcb863e6f959674b9e88693fd576a00a18194e840734874/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b2efaa8-4e8a-4f95-b766-6331405d9856/volumes/kubernetes.io~projected/kube-api-access-g8rfz DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f815fdf2-7e68-4622-a30c-0e864a88b779/volumes/kubernetes.io~projected/kube-api-access-wv76w DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6560c25c11b3b6fdb840252496662f3e1b094b1d03d07063f6770a88af0f1d72/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e99b660b-6975-4a0d-819a-1c9118904b50/volumes/kubernetes.io~projected/kube-api-access-bhfxw DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9270f04e-1085-495a-9636-d4cba5958f5b/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:228f656ade4f3cb MacAddress:c2:24:ca:0f:13:db Speed:10000 Mtu:8901} {Name:27cd4538748a2a0 MacAddress:ba:18:13:03:6c:bf Speed:10000 Mtu:8901} {Name:3d01e25af419821 MacAddress:02:77:af:41:6c:09 Speed:10000 Mtu:8901} {Name:9f9d72cd5249e45 MacAddress:26:78:d2:cb:94:6b Speed:10000 Mtu:8901} {Name:ac2bfd9e018f2ef MacAddress:5a:1d:98:8c:8c:7f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cc:7d:16:a1 Speed:0 Mtu:9001} {Name:br-int MacAddress:12:50:26:89:e3:61 Speed:0 Mtu:8901} {Name:c11ab5db0e79af4 MacAddress:ee:e8:b7:0a:2d:85 Speed:10000 Mtu:8901} {Name:c20ed09978b6b00 MacAddress:ba:86:cd:64:1d:95 Speed:10000 Mtu:8901} {Name:dd7b7cedbcf465e MacAddress:f6:b2:ab:c4:90:6e Speed:10000 Mtu:8901} {Name:eb5f378e4ffec66 MacAddress:76:6a:31:78:0f:8a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cc:7d:16:a1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7e:c5:fe:dc:31:9a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:52:7a:e3:a7:dc:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.646943 5209 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.646987 5209 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647259 5209 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647273 5209 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-236.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647390 5209 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647398 5209 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647412 5209 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647429 5209 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647494 5209 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647644 5209 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647757 5209 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647791 5209 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647802 5209 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647813 5209 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647823 5209 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:45:18.648765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.647831 5209 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:45:18.649697 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.649531 5209 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:45:18.649697 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.649548 5209 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:45:18.650215 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.650194 5209 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:45:18.652870 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.652854 5209 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:45:18.652870 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.652868 5209 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:45:18.652977 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.652962 5209 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 18:45:18.653166 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653154 5209 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:45:18.653205 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653184 5209 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:45:18.653277 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653267 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653281 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653287 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653293 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653299 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653305 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:45:18.653313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653311 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653317 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653324 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653330 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653338 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653347 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653375 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:45:18.653466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.653382 5209 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:45:18.656641 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.656628 5209 server.go:1263] "Started kubelet" Nov 03 18:45:18.657044 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.656719 5209 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:45:18.657044 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.656807 5209 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:45:18.657044 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.656882 5209 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:45:18.657209 ip-10-0-133-236 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:45:18.657980 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.657476 5209 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:45:18.658793 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.658774 5209 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:45:18.661679 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.661598 5209 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.661949 5209 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.662246 5209 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.662285 5209 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.662292 5209 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.662305 5209 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:08 +0000 UTC" deadline="2028-04-14 18:41:42.550762657 +0000 UTC" Nov 03 18:45:18.662383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.662323 5209 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12671h56m23.888443782s" Nov 03 18:45:18.662697 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.662571 5209 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:45:18.663776 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.663410 5209 factory.go:55] Registering systemd factory Nov 03 18:45:18.663776 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.663432 5209 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:45:18.665318 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665297 5209 factory.go:153] Registering CRI-O factory Nov 03 18:45:18.665409 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665321 5209 factory.go:223] Registration of the crio container factory successfully Nov 03 18:45:18.665409 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665388 5209 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:45:18.665511 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665416 5209 factory.go:103] Registering Raw factory Nov 03 18:45:18.665511 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665467 5209 manager.go:1199] Started watching for new ooms in manager Nov 03 18:45:18.665511 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665398 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a6de69-0836-498b-bd0e-704551c14e18" volumeName="kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665539 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b2efaa8-4e8a-4f95-b766-6331405d9856" volumeName="kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665570 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665595 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c661c7b5-69d3-4001-8552-ac76facdb2d5" volumeName="kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665611 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99b660b-6975-4a0d-819a-1c9118904b50" volumeName="kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665624 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665656 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle" seLinuxMountContext="" Nov 03 18:45:18.665673 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665675 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008845af-8b96-4e32-acca-071973319800" volumeName="kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665701 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12643498-998e-4478-9045-6c384c87b101" volumeName="kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665716 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36e2360c-6ea0-40aa-835d-dc764716cd98" volumeName="kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665737 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="979d9146-3563-412a-8fe7-957dd49031aa" volumeName="kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665753 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a40ce628-5272-4a70-a626-a48e7f83059b" volumeName="kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665769 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665788 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665806 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665827 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665842 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665863 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665881 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665892 5209 manager.go:319] Starting recovery of all containers Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.665892 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666006 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666060 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5fdd90b-d6c2-433a-a278-c8d3da866011" volumeName="kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666081 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b2efaa8-4e8a-4f95-b766-6331405d9856" volumeName="kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz" seLinuxMountContext="" Nov 03 18:45:18.666142 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666103 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fb78b5f-402b-40c9-9378-3eda2ddec2b9" volumeName="kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666146 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666176 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a6de69-0836-498b-bd0e-704551c14e18" volumeName="kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666215 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8183d10b-0438-45e9-8e48-02c1cd3cc9f5" volumeName="kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666230 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666249 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666258 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a6de69-0836-498b-bd0e-704551c14e18" volumeName="kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666267 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666291 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666343 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666360 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666370 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a40ce628-5272-4a70-a626-a48e7f83059b" volumeName="kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666378 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666389 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12643498-998e-4478-9045-6c384c87b101" volumeName="kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666399 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666410 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12643498-998e-4478-9045-6c384c87b101" volumeName="kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666418 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fb78b5f-402b-40c9-9378-3eda2ddec2b9" volumeName="kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666427 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="979d9146-3563-412a-8fe7-957dd49031aa" volumeName="kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666458 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a6de69-0836-498b-bd0e-704551c14e18" volumeName="kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666477 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666492 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9609bd5a-37de-4bb8-921b-49c5885e4817" volumeName="kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666500 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666511 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f6b64e-a198-460e-9c07-3eba0cc90e9a" volumeName="kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666526 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36e2360c-6ea0-40aa-835d-dc764716cd98" volumeName="kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666537 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b32ce1b3-62ad-4273-8763-997294b83b96" volumeName="kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666549 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5fdd90b-d6c2-433a-a278-c8d3da866011" volumeName="kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666558 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99b660b-6975-4a0d-819a-1c9118904b50" volumeName="kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666574 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12643498-998e-4478-9045-6c384c87b101" volumeName="kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666587 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="979d9146-3563-412a-8fe7-957dd49031aa" volumeName="kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666595 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="979d9146-3563-412a-8fe7-957dd49031aa" volumeName="kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666607 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f815fdf2-7e68-4622-a30c-0e864a88b779" volumeName="kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666616 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9270f04e-1085-495a-9636-d4cba5958f5b" volumeName="kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666631 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99b660b-6975-4a0d-819a-1c9118904b50" volumeName="kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666666 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008845af-8b96-4e32-acca-071973319800" volumeName="kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666676 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12643498-998e-4478-9045-6c384c87b101" volumeName="kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666689 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a40ce628-5272-4a70-a626-a48e7f83059b" volumeName="kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666698 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9cc12e-64ec-4536-8e8d-02f28afcd385" volumeName="kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666707 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d9666f-95f8-4f02-b87f-1af832cce56f" volumeName="kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666721 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d9666f-95f8-4f02-b87f-1af832cce56f" volumeName="kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666731 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f6b64e-a198-460e-9c07-3eba0cc90e9a" volumeName="kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666742 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99b660b-6975-4a0d-819a-1c9118904b50" volumeName="kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666762 5209 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5fdd90b-d6c2-433a-a278-c8d3da866011" volumeName="kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy" seLinuxMountContext="" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666772 5209 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:45:18.667941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.666777 5209 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:45:18.688903 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.688855 5209 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:45:18.690451 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.690427 5209 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:45:18.690564 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.690553 5209 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:45:18.690886 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.690868 5209 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:45:18.690964 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.690923 5209 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 18:45:18.725849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.725809 5209 generic.go:356] "Generic (PLEG): container finished" podID="b32ce1b3-62ad-4273-8763-997294b83b96" containerID="7127d92423f3776f4b048d1156d8fee939c32a6d1df87be4091af472f883c722" exitCode=0 Nov 03 18:45:18.734047 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734024 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="969190d7abc9667a5646a2c5040e80de24a95c453b6e152460cb74eaae209ae7" exitCode=0 Nov 03 18:45:18.734047 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734044 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="f91ccf5a49f300c171ac6cdd42d8b965cbdfb66120d67d969d0e67d9031fbb4d" exitCode=0 Nov 03 18:45:18.734047 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734050 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="d11a7369fe2d71bd5086c7d798ff771a26de1761b0a8d7d69b1c71e7f137924b" exitCode=0 Nov 03 18:45:18.734279 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734057 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="9dee30c23539089d1d6cf849af4a95ffe5932aac358bb6dd338503f0daac4dcd" exitCode=0 Nov 03 18:45:18.734279 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734062 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="c9df72e7bfc01ee95ebde1c6cb66e645b817ca56a45ba786ee1fd5804f41d29a" exitCode=0 Nov 03 18:45:18.734279 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.734068 5209 generic.go:356] "Generic (PLEG): container finished" podID="e99b660b-6975-4a0d-819a-1c9118904b50" containerID="8ca229ee2eed349defc77fef1c3245df687446f7cbdcf3115399ce2a7f069702" exitCode=0 Nov 03 18:45:18.737437 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.737413 5209 generic.go:356] "Generic (PLEG): container finished" podID="c1266a5d4aae1fd29adb6e6003e98506" containerID="b13ab6f55abab0aa6c74c64e78bba44564226f615ede6a04ccec4ebea440531c" exitCode=0 Nov 03 18:45:18.757984 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.757967 5209 manager.go:324] Recovery completed Nov 03 18:45:18.762311 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.762288 5209 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:45:18.776369 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.776351 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.778198 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.778180 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.778278 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.778215 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.778278 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.778230 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.780650 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780636 5209 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:45:18.780650 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780648 5209 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:45:18.780731 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780657 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.780731 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780665 5209 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:45:18.780794 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780777 5209 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 18:45:18.780827 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780795 5209 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 18:45:18.780827 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780815 5209 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 18:45:18.780827 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780822 5209 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 18:45:18.780827 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780827 5209 policy_none.go:50] "Start" Nov 03 18:45:18.780975 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780835 5209 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:45:18.780975 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.780845 5209 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:45:18.781038 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781027 5209 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 18:45:18.781067 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781039 5209 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 18:45:18.781067 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781045 5209 policy_none.go:44] "Start" Nov 03 18:45:18.781470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781454 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.781525 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781482 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.781525 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.781492 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.785430 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.785406 5209 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:45:18.787908 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.787873 5209 manager.go:341] "Starting Device Plugin manager" Nov 03 18:45:18.787908 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.787896 5209 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:45:18.787908 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.787904 5209 server.go:87] "Starting device plugin registration server" Nov 03 18:45:18.788229 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788217 5209 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:45:18.788264 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788232 5209 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:45:18.788331 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788314 5209 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:45:18.788458 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788421 5209 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:45:18.788550 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788462 5209 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:45:18.788550 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788470 5209 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:45:18.788652 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.788516 5209 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:45:18.788452962Z","UUID":"a3f4428b-c386-4981-82e5-852364352cc9","Handler":null,"Name":"","Endpoint":""} Nov 03 18:45:18.789848 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.789833 5209 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:45:18.789929 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.789855 5209 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:45:18.790062 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.790044 5209 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:45:18.790112 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.790082 5209 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" Nov 03 18:45:18.791110 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.791073 5209 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal"] Nov 03 18:45:18.791251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.791237 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.792061 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.792042 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.792161 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.792066 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.792161 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.792080 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.792383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.792366 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.793177 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.793159 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.793993 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.793973 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.794081 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794005 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.794081 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794024 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.794460 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794438 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.794552 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794469 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.794552 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794485 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.794651 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794569 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"0683633c577f920552f48984254eada9","Type":"ContainerStarted","Data":"dce109b7fc0ecaabd57977e2e3f62bb5e338b82e64fb0b3340b8293d69f1760b"} Nov 03 18:45:18.794704 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794626 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"0683633c577f920552f48984254eada9","Type":"ContainerStarted","Data":"703ecea9d729f56c8324a1997ee4a07a22fa333ce66bb36eb6fafee735e78b73"} Nov 03 18:45:18.794756 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.794671 5209 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.794756 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794714 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerStarted","Data":"1e1133620545e4f9f62d41a4962bbf5251f680c8fd2647ef14430eb457d0d82b"} Nov 03 18:45:18.794756 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794738 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerDied","Data":"b13ab6f55abab0aa6c74c64e78bba44564226f615ede6a04ccec4ebea440531c"} Nov 03 18:45:18.794885 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794765 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" event={"ID":"c1266a5d4aae1fd29adb6e6003e98506","Type":"ContainerStarted","Data":"68bf38204369243f521fe73d9732f69c35ea62ea2abf4d0541a408c9bd119e77"} Nov 03 18:45:18.794935 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.794884 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.795799 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.795773 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.795886 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.795812 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.795886 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.795828 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.796016 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.795996 5209 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.us-west-2.compute.internal\" not found" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.851293 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.851271 5209 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:45:18.851395 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.851321 5209 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 18:45:18.851395 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.851319 5209 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:45:18.851395 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.851332 5209 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:45:18.862404 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.862384 5209 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.888593 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.888575 5209 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:45:18.889478 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.889464 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:45:18.889551 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.889491 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:45:18.889551 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.889501 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:45:18.889551 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.889537 5209 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.944231 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.944207 5209 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:45:18.944311 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:18.944261 5209 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.944311 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.944278 5209 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.944385 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.944278 5209 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:45:18.951712 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.951684 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.951798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.951726 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:18.951798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:18.951755 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.000248 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.000230 5209 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.000299 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.000287 5209 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.000805 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.000774 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeNotReady" Nov 03 18:45:19.000876 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.000814 5209 setters.go:546] "Node became not ready" node="ip-10-0-133-236.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T18:45:19Z","lastTransitionTime":"2026-11-03T18:45:19Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 18:45:19.003739 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.003720 5209 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:45:19.003822 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:19.003756 5209 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052025 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.051961 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052025 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.051966 5209 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-236.us-west-2.compute.internal" is owned by different node (oldNodeID="a455977c-7443-4d33-9968-cecbbe41e732", newNodeID=""), cleaning up... Nov 03 18:45:19.052025 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.052017 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0683633c577f920552f48984254eada9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"0683633c577f920552f48984254eada9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.052063 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.052084 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.052135 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.052216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.052171 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1266a5d4aae1fd29adb6e6003e98506-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" (UID: \"c1266a5d4aae1fd29adb6e6003e98506\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.649083 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.649056 5209 apiserver.go:52] "Watching apiserver" Nov 03 18:45:19.742623 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.742598 5209 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.775745 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.775722 5209 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:45:19.776232 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.776213 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-z9xv5","openshift-monitoring/metrics-server-85778475ff-j2r6b","openshift-monitoring/node-exporter-4shpc","openshift-network-diagnostics/network-check-target-dbs68","openshift-dns/node-resolver-pxj4z","openshift-ingress-canary/ingress-canary-j6lk6","openshift-dns/dns-default-hlbp5","openshift-multus/multus-additional-cni-plugins-75kw7","openshift-ovn-kubernetes/ovnkube-node-ndlb6","kube-system/global-pull-secret-syncer-x7248","openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr","openshift-network-operator/iptables-alerter-dvjcd","openshift-multus/network-metrics-daemon-gvgvl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk","openshift-insights/insights-runtime-extractor-2c66j","openshift-multus/multus-222xl","openshift-cluster-node-tuning-operator/tuned-js4mm","kube-system/konnectivity-agent-xnctw","kube-system/kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal","openshift-monitoring/telemeter-client-6c7dc9f994-29p67","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal"] Nov 03 18:45:19.799408 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.799386 5209 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:45:19.799505 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:19.799424 5209 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.us-west-2.compute.internal" Nov 03 18:45:19.826717 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.826695 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\"" Nov 03 18:45:19.826717 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.826709 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\"" Nov 03 18:45:19.826852 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.826695 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:45:19.826852 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.826829 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:45:19.827255 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.827241 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:45:19.864557 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.864537 5209 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:45:19.927256 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927175 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:45:19.927576 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927550 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 18:45:19.927576 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927551 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 18:45:19.927762 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927602 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:45:19.927822 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927770 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\"" Nov 03 18:45:19.927947 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.927933 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\"" Nov 03 18:45:19.928111 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928095 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\"" Nov 03 18:45:19.928241 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928224 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:45:19.928399 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928376 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\"" Nov 03 18:45:19.928494 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928399 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:45:19.928590 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928576 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltsf8\"" Nov 03 18:45:19.928653 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928604 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zgffh\"" Nov 03 18:45:19.928653 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928645 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 18:45:19.928738 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928670 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 18:45:19.928773 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928747 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 18:45:19.928929 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928914 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:45:19.928984 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928957 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:45:19.928984 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.928962 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.929070 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929015 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\"" Nov 03 18:45:19.929070 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929034 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929070 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929046 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\"" Nov 03 18:45:19.929070 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929068 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929054 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:45:19.929258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929069 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929274 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929261 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929294 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929304 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929308 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929324 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929338 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929350 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929350 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929364 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929352 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929396 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929409 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929464 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929471 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929473 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dz7cd\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929487 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929497 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929503 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929561 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929571 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:45:19.929646 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929591 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929654 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929660 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929656 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929658 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929681 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929712 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929738 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929747 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929781 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929785 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929800 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2o4ukbnja9n82\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929812 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929817 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929781 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929846 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929811 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929846 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929870 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929847 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mrvdd\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929858 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929878 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929885 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929905 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929909 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.929932 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.930022 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.930075 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:45:19.930129 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.930112 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 18:45:19.956398 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956380 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.956446 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956401 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.956446 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956421 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.956446 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956438 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.956560 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956455 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.956560 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956514 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.956560 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956556 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.956691 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956643 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.956691 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956664 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:45:19.956691 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956680 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956727 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956739 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-config\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956754 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956774 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956776 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-accelerators-collector-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956803 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-textfile\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.956844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956813 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956861 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtlq\" (UniqueName: \"kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956907 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956929 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956954 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956976 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8gr\" (UniqueName: \"kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956991 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957009 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.956977 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/008845af-8b96-4e32-acca-071973319800-cert\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957032 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957089 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957168 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv76w\" (UniqueName: \"kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.957181 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957179 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957196 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957242 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957253 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-tmp\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957272 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957275 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-original-pull-secret\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957296 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957333 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957362 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957379 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957399 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-client-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957423 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lm9v\" (UniqueName: \"kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957503 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957548 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957572 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957589 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957618 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957670 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:19.957744 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957738 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957757 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957850 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36e2360c-6ea0-40aa-835d-dc764716cd98-agent-certs\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957856 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjgf\" (UniqueName: \"kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957872 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12643498-998e-4478-9045-6c384c87b101-ovn-node-metrics-cert\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957889 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957918 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957926 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2efaa8-4e8a-4f95-b766-6331405d9856-metrics-certs\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957946 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957974 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957997 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a40ce628-5272-4a70-a626-a48e7f83059b-etc-tuned\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.957998 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958030 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfxw\" (UniqueName: \"kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958050 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958071 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958087 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7dr\" (UniqueName: \"kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958103 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958139 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9a6de69-0836-498b-bd0e-704551c14e18-metrics-tls\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958175 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958199 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/979d9146-3563-412a-8fe7-957dd49031aa-data-volume\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958213 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958246 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958283 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72g86\" (UniqueName: \"kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958318 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958319 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-cni-binary-copy\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958331 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-cni-binary-copy\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958356 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958385 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958410 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958451 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxlz\" (UniqueName: \"kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.958470 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958479 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958506 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958532 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-metrics-client-ca\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958542 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958601 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958635 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958651 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-tls\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958687 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-daemon-config\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958712 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958732 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tsj\" (UniqueName: \"kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958749 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958762 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958772 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958780 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjrt\" (UniqueName: \"kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958795 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958839 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9cl\" (UniqueName: \"kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958869 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-tls\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958885 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-iptables-alerter-script\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958911 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958949 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958975 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.958997 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959021 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959046 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959077 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959104 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959148 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-ovnkube-script-lib\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959141 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvp5\" (UniqueName: \"kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959186 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959196 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959224 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq8w\" (UniqueName: \"kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959238 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/979d9146-3563-412a-8fe7-957dd49031aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959249 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959275 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959302 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12643498-998e-4478-9045-6c384c87b101-env-overrides\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959302 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959351 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959372 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36e2360c-6ea0-40aa-835d-dc764716cd98-konnectivity-ca\") pod \"konnectivity-agent-xnctw\" (UID: \"36e2360c-6ea0-40aa-835d-dc764716cd98\") " pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959372 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959403 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959438 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/979d9146-3563-412a-8fe7-957dd49031aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959476 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959531 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-serviceca\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959536 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a6de69-0836-498b-bd0e-704551c14e18-tmp-dir\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959584 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959615 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959649 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.959660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959681 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959697 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959719 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959736 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959747 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99b660b-6975-4a0d-819a-1c9118904b50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959779 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959799 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959835 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959864 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959870 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f815fdf2-7e68-4622-a30c-0e864a88b779-secret-metrics-server-client-certs\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959899 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959934 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959951 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ad9cc12e-64ec-4536-8e8d-02f28afcd385-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959964 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rfz\" (UniqueName: \"kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959954 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.959999 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960007 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f815fdf2-7e68-4622-a30c-0e864a88b779-audit-log\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960090 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddbr\" (UniqueName: \"kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960144 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960190 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960219 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960258 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960294 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960309 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-serving-certs-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960337 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960355 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960371 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960386 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960400 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960416 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960431 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a6de69-0836-498b-bd0e-704551c14e18-config-volume\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960442 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960478 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960500 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960514 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d9666f-95f8-4f02-b87f-1af832cce56f-tmp-dir\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960521 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad9cc12e-64ec-4536-8e8d-02f28afcd385-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960527 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960569 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960597 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960600 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b32ce1b3-62ad-4273-8763-997294b83b96-metrics-client-ca\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960624 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960673 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960695 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960700 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960751 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9270f04e-1085-495a-9636-d4cba5958f5b-federate-client-tls\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:19.961258 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.960686 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f815fdf2-7e68-4622-a30c-0e864a88b779-metrics-server-audit-profiles\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:19.974738 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.974705 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 18:45:19.980972 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:19.980949 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9270f04e-1085-495a-9636-d4cba5958f5b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:20.012856 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.012830 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtlq\" (UniqueName: \"kubernetes.io/projected/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-kube-api-access-gvtlq\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:20.012987 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.012966 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lm9v\" (UniqueName: \"kubernetes.io/projected/e5fdd90b-d6c2-433a-a278-c8d3da866011-kube-api-access-9lm9v\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.013052 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013014 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8gr\" (UniqueName: \"kubernetes.io/projected/e9a6de69-0836-498b-bd0e-704551c14e18-kube-api-access-pn8gr\") pod \"dns-default-hlbp5\" (UID: \"e9a6de69-0836-498b-bd0e-704551c14e18\") " pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:20.013280 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013264 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j8q\" (UniqueName: \"kubernetes.io/projected/9609bd5a-37de-4bb8-921b-49c5885e4817-kube-api-access-h4j8q\") pod \"network-check-target-dbs68\" (UID: \"9609bd5a-37de-4bb8-921b-49c5885e4817\") " pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:20.013330 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013277 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv76w\" (UniqueName: \"kubernetes.io/projected/f815fdf2-7e68-4622-a30c-0e864a88b779-kube-api-access-wv76w\") pod \"metrics-server-85778475ff-j2r6b\" (UID: \"f815fdf2-7e68-4622-a30c-0e864a88b779\") " pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:20.013555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013532 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72g86\" (UniqueName: \"kubernetes.io/projected/b6d9666f-95f8-4f02-b87f-1af832cce56f-kube-api-access-72g86\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:20.013703 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013682 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjgf\" (UniqueName: \"kubernetes.io/projected/008845af-8b96-4e32-acca-071973319800-kube-api-access-dmjgf\") pod \"ingress-canary-j6lk6\" (UID: \"008845af-8b96-4e32-acca-071973319800\") " pod="openshift-ingress-canary/ingress-canary-j6lk6" Nov 03 18:45:20.013804 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013789 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxlz\" (UniqueName: \"kubernetes.io/projected/b32ce1b3-62ad-4273-8763-997294b83b96-kube-api-access-tjxlz\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.013856 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.013820 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfxw\" (UniqueName: \"kubernetes.io/projected/e99b660b-6975-4a0d-819a-1c9118904b50-kube-api-access-bhfxw\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.014060 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.014039 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7dr\" (UniqueName: \"kubernetes.io/projected/ad9cc12e-64ec-4536-8e8d-02f28afcd385-kube-api-access-kn7dr\") pod \"kube-state-metrics-5dc8d7d89b-vznsr\" (UID: \"ad9cc12e-64ec-4536-8e8d-02f28afcd385\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vznsr" Nov 03 18:45:20.014060 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.014053 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tsj\" (UniqueName: \"kubernetes.io/projected/979d9146-3563-412a-8fe7-957dd49031aa-kube-api-access-f9tsj\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:20.014640 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.014621 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvp5\" (UniqueName: \"kubernetes.io/projected/9270f04e-1085-495a-9636-d4cba5958f5b-kube-api-access-fcvp5\") pod \"telemeter-client-6c7dc9f994-29p67\" (UID: \"9270f04e-1085-495a-9636-d4cba5958f5b\") " pod="openshift-monitoring/telemeter-client-6c7dc9f994-29p67" Nov 03 18:45:20.014751 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.014737 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq8w\" (UniqueName: \"kubernetes.io/projected/c661c7b5-69d3-4001-8552-ac76facdb2d5-kube-api-access-8pq8w\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.015671 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.015655 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjrt\" (UniqueName: \"kubernetes.io/projected/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-kube-api-access-cgjrt\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:20.019843 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.019824 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rfz\" (UniqueName: \"kubernetes.io/projected/2b2efaa8-4e8a-4f95-b766-6331405d9856-kube-api-access-g8rfz\") pod \"network-metrics-daemon-gvgvl\" (UID: \"2b2efaa8-4e8a-4f95-b766-6331405d9856\") " pod="openshift-multus/network-metrics-daemon-gvgvl" Nov 03 18:45:20.020038 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.020018 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddbr\" (UniqueName: \"kubernetes.io/projected/a40ce628-5272-4a70-a626-a48e7f83059b-kube-api-access-bddbr\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.020178 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.020165 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9cl\" (UniqueName: \"kubernetes.io/projected/12643498-998e-4478-9045-6c384c87b101-kube-api-access-hn9cl\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061212 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061191 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.061286 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061217 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061286 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061231 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061286 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061245 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061286 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061258 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061286 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061276 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061288 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061312 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-systemd-units\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061325 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061319 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-node-log\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061350 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-etc-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061354 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061380 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-bin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061388 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061401 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061399 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-conf\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061416 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061423 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysctl-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061420 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061437 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-multus-certs\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061452 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061469 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061500 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f6b64e-a198-460e-9c07-3eba0cc90e9a-host\") pod \"node-ca-z9xv5\" (UID: \"e7f6b64e-a198-460e-9c07-3eba0cc90e9a\") " pod="openshift-image-registry/node-ca-z9xv5" Nov 03 18:45:20.061507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061509 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061522 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061544 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061566 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061580 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061581 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-device-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061611 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061616 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061633 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061649 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061666 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061683 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061697 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-lib-modules\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061710 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-kubelet\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061718 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-var-lib-openvswitch\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061731 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061742 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-bin\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061753 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-modprobe-d\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061748 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061764 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-netns\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061763 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061779 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061783 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-etc-kubernetes\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061799 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-kubelet-config\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061810 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061815 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-systemd\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061834 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061852 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061854 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-registration-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061866 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061881 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061883 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-kubernetes\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061895 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-sys\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061907 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-sys-fs\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061902 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061918 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-etc-sysconfig\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061930 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-systemd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061934 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061958 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061973 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.061986 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062001 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062013 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-netns\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062024 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062041 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-hostroot\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062040 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-cni-netd\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062051 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062065 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-conf-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062053 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-host\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062075 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.062041 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062052 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-var-lib-kubelet\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062085 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-sys\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062091 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062106 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062135 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062142 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-run-ovn\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062151 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062184 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-kubelet\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062187 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/979d9146-3563-412a-8fe7-957dd49031aa-crio-socket\") pod \"insights-runtime-extractor-2c66j\" (UID: \"979d9146-3563-412a-8fe7-957dd49031aa\") " pod="openshift-insights/insights-runtime-extractor-2c66j" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062199 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062214 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062218 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c661c7b5-69d3-4001-8552-ac76facdb2d5-socket-dir\") pod \"aws-ebs-csi-driver-node-7p7vk\" (UID: \"c661c7b5-69d3-4001-8552-ac76facdb2d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7p7vk" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062229 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062249 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-run-k8s-cni-cncf-io\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062250 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062230 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-system-cni-dir\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062273 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a40ce628-5272-4a70-a626-a48e7f83059b-run\") pod \"tuned-js4mm\" (UID: \"a40ce628-5272-4a70-a626-a48e7f83059b\") " pod="openshift-cluster-node-tuning-operator/tuned-js4mm" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062280 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-run-ovn-kubernetes\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062288 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-system-cni-dir\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062273 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062322 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062327 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-cnibin\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062341 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062356 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062371 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062382 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99b660b-6975-4a0d-819a-1c9118904b50-cnibin\") pod \"multus-additional-cni-plugins-75kw7\" (UID: \"e99b660b-6975-4a0d-819a-1c9118904b50\") " pod="openshift-multus/multus-additional-cni-plugins-75kw7" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062391 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062416 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fb78b5f-402b-40c9-9378-3eda2ddec2b9-host-slash\") pod \"iptables-alerter-dvjcd\" (UID: \"8fb78b5f-402b-40c9-9378-3eda2ddec2b9\") " pod="openshift-network-operator/iptables-alerter-dvjcd" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062428 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6d9666f-95f8-4f02-b87f-1af832cce56f-hosts-file\") pod \"node-resolver-pxj4z\" (UID: \"b6d9666f-95f8-4f02-b87f-1af832cce56f\") " pod="openshift-dns/node-resolver-pxj4z" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062434 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-multus-socket-dir-parent\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062414 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062454 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-host-slash\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062478 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062493 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062509 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062525 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12643498-998e-4478-9045-6c384c87b101-log-socket\") pod \"ovnkube-node-ndlb6\" (UID: \"12643498-998e-4478-9045-6c384c87b101\") " pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062547 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5fdd90b-d6c2-433a-a278-c8d3da866011-host-var-lib-cni-multus\") pod \"multus-222xl\" (UID: \"e5fdd90b-d6c2-433a-a278-c8d3da866011\") " pod="openshift-multus/multus-222xl" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062559 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-root\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062562 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8183d10b-0438-45e9-8e48-02c1cd3cc9f5-dbus\") pod \"global-pull-secret-syncer-x7248\" (UID: \"8183d10b-0438-45e9-8e48-02c1cd3cc9f5\") " pod="kube-system/global-pull-secret-syncer-x7248" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062564 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.063570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.062595 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b32ce1b3-62ad-4273-8763-997294b83b96-node-exporter-wtmp\") pod \"node-exporter-4shpc\" (UID: \"b32ce1b3-62ad-4273-8763-997294b83b96\") " pod="openshift-monitoring/node-exporter-4shpc" Nov 03 18:45:20.988804 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:20.988767 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hlbp5" Nov 03 18:45:23.362094 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:23.362060 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnctw" Nov 03 18:45:24.354104 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:24.354073 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:25.931555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:25.931524 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:26.897131 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:26.897090 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:26.897368 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:26.897351 5209 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:45:26.911710 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:26.911681 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ndlb6" Nov 03 18:45:29.118070 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:29.117992 5209 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-236.us-west-2.compute.internal" event="NodeReady" Nov 03 18:45:31.935367 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:45:31.935332 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:45:42.451602 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:42.451568 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85778475ff-j2r6b" Nov 03 18:45:53.420428 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.420397 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:45:53.424080 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.424063 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.460816 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.460788 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.460952 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.460829 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.460952 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.460862 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461064 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.460975 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461135 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461057 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4jm\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-kube-api-access-lx4jm\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461204 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461135 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461204 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461167 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461204 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461198 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461302 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461334 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461351 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461374 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.461420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.461390 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.467257 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.467235 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:45:53.502725 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502704 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 18:45:53.502830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502741 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 18:45:53.502830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502752 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 18:45:53.502830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502767 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 18:45:53.502830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502772 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 18:45:53.502830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502790 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 18:45:53.502991 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.502853 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ltsw\"" Nov 03 18:45:53.504309 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.504296 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 18:45:53.504351 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.504316 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 18:45:53.561748 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561728 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561819 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561757 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561819 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561776 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4jm\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-kube-api-access-lx4jm\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561819 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561806 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561938 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561833 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561938 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561860 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.561938 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561900 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562076 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561937 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562076 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.561965 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562076 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562000 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562088 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562166 5209 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:45:53.562251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562178 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562228 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562525 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.562969 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.562901 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565055 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565010 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565164 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565144 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565397 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565367 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565482 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565398 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565482 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565422 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565482 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565425 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565695 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565678 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.565766 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.565723 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.566645 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.566626 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.568917 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.568899 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 18:45:53.573601 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.573581 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.621064 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.621034 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4jm\" (UniqueName: \"kubernetes.io/projected/6368e126-6e6c-43e6-a415-6ce9f87bfcb9-kube-api-access-lx4jm\") pod \"alertmanager-main-0\" (UID: \"6368e126-6e6c-43e6-a415-6ce9f87bfcb9\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:53.739722 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:53.739650 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:45:54.219028 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:54.218996 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:45:54.226074 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:45:54.226042 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6368e126_6e6c_43e6_a415_6ce9f87bfcb9.slice/crio-578a7ef56ca45b49e5ccf9493af74bb112ed59ee472119119fd92716513b17bd WatchSource:0}: Error finding container 578a7ef56ca45b49e5ccf9493af74bb112ed59ee472119119fd92716513b17bd: Status 404 returned error can't find the container with id 578a7ef56ca45b49e5ccf9493af74bb112ed59ee472119119fd92716513b17bd Nov 03 18:45:54.849321 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:54.849284 5209 generic.go:356] "Generic (PLEG): container finished" podID="6368e126-6e6c-43e6-a415-6ce9f87bfcb9" containerID="d789f4b566d362219ed58879a708a65e7a5b727e95cf9f7e0bfe25d1470c320b" exitCode=0 Nov 03 18:45:54.849768 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:54.849326 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerDied","Data":"d789f4b566d362219ed58879a708a65e7a5b727e95cf9f7e0bfe25d1470c320b"} Nov 03 18:45:54.849768 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:54.849393 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"578a7ef56ca45b49e5ccf9493af74bb112ed59ee472119119fd92716513b17bd"} Nov 03 18:45:54.850655 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:54.850638 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:45:55.934346 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:55.934318 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dbs68" Nov 03 18:45:57.863057 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:57.863023 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"26738ddb83dc5637b339f7bb57630ecb8893954b4bfbf867d3d24d44cd3d2658"} Nov 03 18:45:57.863435 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:57.863140 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"a6a3844e377d70f0e4b40ebf08428d0e737b8c2c06197b7710e77d641e640c55"} Nov 03 18:45:57.863435 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:57.863168 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"15f6933596ae4e2eb13e00e26a5b2dad0ab9b3ca5382cd015fadf60726a2d48b"} Nov 03 18:45:57.863435 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:57.863192 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"c5b4fa98426e13c92899ea8cd9b9d3a59bb56278e4cc44e8dd66d9c79007caa0"} Nov 03 18:45:57.863435 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:57.863217 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"b073e0c357cfeb6dc5656dcb7eaa22a51e110bbacfb84b03c483c9ec65ad6f0e"} Nov 03 18:45:59.872798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:45:59.872756 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6368e126-6e6c-43e6-a415-6ce9f87bfcb9","Type":"ContainerStarted","Data":"7fa0fc2581009bd361df1e52e92eda41dfd5702b7733ac1260fd9e410c8c7104"} Nov 03 18:46:00.134387 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:46:00.134275 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.403760764 podStartE2EDuration="7.134262486s" totalImagesPullingTime="3.71882868s" totalInitContainerRuntime="11.673042ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:00.087611226 +0000 UTC m=+41.498711349" watchObservedRunningTime="2026-11-03 18:46:00.134262486 +0000 UTC m=+41.545362605" Nov 03 18:46:01.936252 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:46:01.936217 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:46:31.934999 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:46:31.934964 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:47:01.935717 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:47:01.935635 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:47:21.157461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.157425 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q"] Nov 03 18:47:21.160618 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.160602 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.180916 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.180879 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.181047 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.180931 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.181047 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.180987 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrbr\" (UniqueName: \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.256344 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.256306 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 18:47:21.256513 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.256349 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 18:47:21.256513 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.256381 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-7fkhb\"" Nov 03 18:47:21.257016 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.256998 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q"] Nov 03 18:47:21.281555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.281529 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.281639 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.281570 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.281639 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.281601 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrbr\" (UniqueName: \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.281993 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.281972 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.282069 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.281977 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.339770 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrbr\" (UniqueName: \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.475804 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.475730 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:21.841132 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:21.841100 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q"] Nov 03 18:47:21.847537 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:47:21.847506 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0802e94a_e7b5_4ece_9e59_14fc699e10ac.slice/crio-e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f WatchSource:0}: Error finding container e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f: Status 404 returned error can't find the container with id e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f Nov 03 18:47:22.127996 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:22.127916 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerStarted","Data":"e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f"} Nov 03 18:47:28.149705 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:28.149673 5209 generic.go:356] "Generic (PLEG): container finished" podID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerID="c6a6ee44f74ad93e20535d6a9a858febcb55b08a61998860c009842e12ae235e" exitCode=0 Nov 03 18:47:28.150107 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:28.149731 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerDied","Data":"c6a6ee44f74ad93e20535d6a9a858febcb55b08a61998860c009842e12ae235e"} Nov 03 18:47:31.936489 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:47:31.936450 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:47:32.166105 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:32.166072 5209 generic.go:356] "Generic (PLEG): container finished" podID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerID="e82b5b33c3463b955355ac96827621e5849d8018571d11d03c110d3916c2beaa" exitCode=0 Nov 03 18:47:32.166284 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:32.166164 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerDied","Data":"e82b5b33c3463b955355ac96827621e5849d8018571d11d03c110d3916c2beaa"} Nov 03 18:47:39.194919 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:39.194875 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerStarted","Data":"cb02dce439e8092999db60491b0d917066f66ffe619ee05ce34a6821b296c84e"} Nov 03 18:47:39.306097 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:39.306043 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" podStartSLOduration=2.893039217 podStartE2EDuration="18.306030725s" totalImagesPullingTime="15.412991508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:39.305603113 +0000 UTC m=+140.716703256" watchObservedRunningTime="2026-11-03 18:47:39.306030725 +0000 UTC m=+140.717130842" Nov 03 18:47:40.199776 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:40.199738 5209 generic.go:356] "Generic (PLEG): container finished" podID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerID="cb02dce439e8092999db60491b0d917066f66ffe619ee05ce34a6821b296c84e" exitCode=0 Nov 03 18:47:40.200192 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:40.199820 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerDied","Data":"cb02dce439e8092999db60491b0d917066f66ffe619ee05ce34a6821b296c84e"} Nov 03 18:47:41.316303 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.316285 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:41.447755 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.447727 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\") pod \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " Nov 03 18:47:41.447918 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.447767 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\") pod \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " Nov 03 18:47:41.447918 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.447816 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\" (UniqueName: \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\") pod \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\" (UID: \"0802e94a-e7b5-4ece-9e59-14fc699e10ac\") " Nov 03 18:47:41.448545 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.448508 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle" pod "0802e94a-e7b5-4ece-9e59-14fc699e10ac" (UID: "0802e94a-e7b5-4ece-9e59-14fc699e10ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:47:41.449976 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.449947 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr" pod "0802e94a-e7b5-4ece-9e59-14fc699e10ac" (UID: "0802e94a-e7b5-4ece-9e59-14fc699e10ac"). InnerVolumeSpecName "kube-api-access-zbrbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:47:41.452660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.452636 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util" pod "0802e94a-e7b5-4ece-9e59-14fc699e10ac" (UID: "0802e94a-e7b5-4ece-9e59-14fc699e10ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:47:41.549159 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.549067 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:41.549159 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.549092 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0802e94a-e7b5-4ece-9e59-14fc699e10ac-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:41.549159 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:41.549102 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrbr\" (UniqueName: \"kubernetes.io/projected/0802e94a-e7b5-4ece-9e59-14fc699e10ac-kube-api-access-zbrbr\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:42.208131 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:42.208093 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" event={"ID":"0802e94a-e7b5-4ece-9e59-14fc699e10ac","Type":"ContainerDied","Data":"e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f"} Nov 03 18:47:42.208293 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:42.208148 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvml9q" Nov 03 18:47:42.208293 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:42.208165 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e889acbb1cac8ebeda125257b4e53211011d01e2522ef22a77474efc62129f" Nov 03 18:47:49.431272 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431239 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x"] Nov 03 18:47:49.431661 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431625 5209 cpu_manager.go:414] "containerMap: removing container" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="pull" Nov 03 18:47:49.431661 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431642 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="pull" Nov 03 18:47:49.431728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431666 5209 cpu_manager.go:414] "containerMap: removing container" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="util" Nov 03 18:47:49.431728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431675 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="util" Nov 03 18:47:49.431728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431685 5209 cpu_manager.go:414] "containerMap: removing container" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="extract" Nov 03 18:47:49.431728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431693 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="extract" Nov 03 18:47:49.431852 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.431754 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="0802e94a-e7b5-4ece-9e59-14fc699e10ac" containerName="extract" Nov 03 18:47:49.433895 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.433880 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.485589 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.485562 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:47:49.485772 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.485676 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 18:47:49.485772 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.485731 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rxdt6\"" Nov 03 18:47:49.509939 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.509910 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq8t\" (UniqueName: \"kubernetes.io/projected/2b54735b-68b0-4ef5-abfe-2b0c9da66992-kube-api-access-qkq8t\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.510036 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.509981 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b54735b-68b0-4ef5-abfe-2b0c9da66992-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.533392 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.533366 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x"] Nov 03 18:47:49.610791 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.610765 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b54735b-68b0-4ef5-abfe-2b0c9da66992-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.610905 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.610807 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq8t\" (UniqueName: \"kubernetes.io/projected/2b54735b-68b0-4ef5-abfe-2b0c9da66992-kube-api-access-qkq8t\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.611201 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.611182 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b54735b-68b0-4ef5-abfe-2b0c9da66992-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.671197 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.671170 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq8t\" (UniqueName: \"kubernetes.io/projected/2b54735b-68b0-4ef5-abfe-2b0c9da66992-kube-api-access-qkq8t\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-ks78x\" (UID: \"2b54735b-68b0-4ef5-abfe-2b0c9da66992\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:49.743363 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:49.743313 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" Nov 03 18:47:50.209714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:50.209684 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x"] Nov 03 18:47:50.216710 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:47:50.216685 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b54735b_68b0_4ef5_abfe_2b0c9da66992.slice/crio-8719499d979fb6b19d2b6ae99eab1b12b5597ecf6c3bd752be3b906a9ad39100 WatchSource:0}: Error finding container 8719499d979fb6b19d2b6ae99eab1b12b5597ecf6c3bd752be3b906a9ad39100: Status 404 returned error can't find the container with id 8719499d979fb6b19d2b6ae99eab1b12b5597ecf6c3bd752be3b906a9ad39100 Nov 03 18:47:50.233589 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:50.233563 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" event={"ID":"2b54735b-68b0-4ef5-abfe-2b0c9da66992","Type":"ContainerStarted","Data":"8719499d979fb6b19d2b6ae99eab1b12b5597ecf6c3bd752be3b906a9ad39100"} Nov 03 18:47:54.250957 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:54.250918 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" event={"ID":"2b54735b-68b0-4ef5-abfe-2b0c9da66992","Type":"ContainerStarted","Data":"748fe16456bc0b57b643ff28fe776f91802124c7a79be0a8279db9b8da6f141c"} Nov 03 18:47:54.457186 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:54.457141 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-ks78x" podStartSLOduration=2.196316316 podStartE2EDuration="5.457113773s" totalImagesPullingTime="3.260797457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:54.410191636 +0000 UTC m=+155.821291778" watchObservedRunningTime="2026-11-03 18:47:54.457113773 +0000 UTC m=+155.868213887" Nov 03 18:47:55.718509 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.718476 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl"] Nov 03 18:47:55.720468 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.720448 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:55.776519 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.776485 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl"] Nov 03 18:47:55.819885 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.819857 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 18:47:55.820028 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.819930 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 18:47:55.820028 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.819948 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-v5bsl\"" Nov 03 18:47:55.864502 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.864464 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:55.864712 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.864671 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46djc\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-kube-api-access-46djc\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:55.965887 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.965856 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:55.966046 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:55.965898 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46djc\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-kube-api-access-46djc\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:56.023891 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.023816 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:56.023891 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.023873 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46djc\" (UniqueName: \"kubernetes.io/projected/90594851-ca2d-495b-bcbd-ed14c297e0b8-kube-api-access-46djc\") pod \"cert-manager-webhook-5dbbbf75cc-g8wjl\" (UID: \"90594851-ca2d-495b-bcbd-ed14c297e0b8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:56.044694 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.044667 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:47:56.095771 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.095741 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh"] Nov 03 18:47:56.098447 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.098423 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.147706 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.147674 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 18:47:56.147872 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.147735 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-46h5w\"" Nov 03 18:47:56.147872 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.147749 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 18:47:56.153802 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.153782 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh"] Nov 03 18:47:56.268593 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.268560 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.268764 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.268601 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7pl\" (UniqueName: \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.268764 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.268622 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.369957 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.369920 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.370114 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.369966 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7pl\" (UniqueName: \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.370114 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.369984 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.370357 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.370340 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.370437 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.370419 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.406735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.406706 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl"] Nov 03 18:47:56.414013 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:47:56.413983 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90594851_ca2d_495b_bcbd_ed14c297e0b8.slice/crio-3243db797c5711a39478fba6d3f1c6095afb5dbe1b2fae6a2eca105f1f371523 WatchSource:0}: Error finding container 3243db797c5711a39478fba6d3f1c6095afb5dbe1b2fae6a2eca105f1f371523: Status 404 returned error can't find the container with id 3243db797c5711a39478fba6d3f1c6095afb5dbe1b2fae6a2eca105f1f371523 Nov 03 18:47:56.427442 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.427416 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7pl\" (UniqueName: \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:56.715318 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:56.715242 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:47:57.165782 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.165748 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh"] Nov 03 18:47:57.174083 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:47:57.174055 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209c4af3_fbd2_4f97_b7cc_fcfd63222186.slice/crio-ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831 WatchSource:0}: Error finding container ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831: Status 404 returned error can't find the container with id ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831 Nov 03 18:47:57.267474 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.267444 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" event={"ID":"90594851-ca2d-495b-bcbd-ed14c297e0b8","Type":"ContainerStarted","Data":"3243db797c5711a39478fba6d3f1c6095afb5dbe1b2fae6a2eca105f1f371523"} Nov 03 18:47:57.269027 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.269004 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerStarted","Data":"dea020ed2475d2dc666c7563ba107a42289ef21ebef330846f26c14d49151fd9"} Nov 03 18:47:57.269145 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.269048 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerStarted","Data":"ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831"} Nov 03 18:47:57.844853 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.844812 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8"] Nov 03 18:47:57.846777 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.846760 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:57.897653 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.897632 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jzdq6\"" Nov 03 18:47:57.902376 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.902347 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8"] Nov 03 18:47:57.981209 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.981179 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:57.981370 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:57.981220 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f792\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-kube-api-access-4f792\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.082157 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.082107 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.082302 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.082173 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f792\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-kube-api-access-4f792\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.140042 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.139978 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f792\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-kube-api-access-4f792\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.140042 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.139992 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fea318f5-9695-4069-8346-302c0693e86e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-j2qg8\" (UID: \"fea318f5-9695-4069-8346-302c0693e86e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.156929 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.156907 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" Nov 03 18:47:58.273789 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.273758 5209 generic.go:356] "Generic (PLEG): container finished" podID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerID="dea020ed2475d2dc666c7563ba107a42289ef21ebef330846f26c14d49151fd9" exitCode=0 Nov 03 18:47:58.274147 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.273818 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerDied","Data":"dea020ed2475d2dc666c7563ba107a42289ef21ebef330846f26c14d49151fd9"} Nov 03 18:47:58.523337 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:58.523252 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8"] Nov 03 18:47:58.530974 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:47:58.530948 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfea318f5_9695_4069_8346_302c0693e86e.slice/crio-20e23df0e4905839ee792bc787d0803a6f280a19f8524a2221151904f5d43ef4 WatchSource:0}: Error finding container 20e23df0e4905839ee792bc787d0803a6f280a19f8524a2221151904f5d43ef4: Status 404 returned error can't find the container with id 20e23df0e4905839ee792bc787d0803a6f280a19f8524a2221151904f5d43ef4 Nov 03 18:47:59.283375 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:47:59.283339 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" event={"ID":"fea318f5-9695-4069-8346-302c0693e86e","Type":"ContainerStarted","Data":"20e23df0e4905839ee792bc787d0803a6f280a19f8524a2221151904f5d43ef4"} Nov 03 18:48:01.939039 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:48:01.939002 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:48:03.302263 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:03.302163 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" event={"ID":"90594851-ca2d-495b-bcbd-ed14c297e0b8","Type":"ContainerStarted","Data":"f9c23dd20861d64210488a4b5f63ef9657a56b3e5369fa4f6fbc06adc49a11bd"} Nov 03 18:48:03.302263 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:03.302243 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:48:03.303775 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:03.303753 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" event={"ID":"fea318f5-9695-4069-8346-302c0693e86e","Type":"ContainerStarted","Data":"76cb1c7a87c0842f9e6ecdb5cafeba53fb1bc025d0d3f34b4e63a357b01f1a62"} Nov 03 18:48:03.411852 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:03.411796 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" podStartSLOduration=1.841120678 podStartE2EDuration="8.411781635s" totalImagesPullingTime="6.570660957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:03.410951682 +0000 UTC m=+164.822051816" watchObservedRunningTime="2026-11-03 18:48:03.411781635 +0000 UTC m=+164.822881755" Nov 03 18:48:03.519926 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:03.519867 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-j2qg8" podStartSLOduration=2.066075425 podStartE2EDuration="6.519846368s" totalImagesPullingTime="4.453770943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:03.519106296 +0000 UTC m=+164.930206439" watchObservedRunningTime="2026-11-03 18:48:03.519846368 +0000 UTC m=+164.930946493" Nov 03 18:48:05.312694 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:05.312656 5209 generic.go:356] "Generic (PLEG): container finished" podID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerID="f70638ad5fa6ce0f67a93f3ddf348a7c22d29a80f6f9e09c97d590adfd144e09" exitCode=0 Nov 03 18:48:05.313084 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:05.312745 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerDied","Data":"f70638ad5fa6ce0f67a93f3ddf348a7c22d29a80f6f9e09c97d590adfd144e09"} Nov 03 18:48:06.318902 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:06.318866 5209 generic.go:356] "Generic (PLEG): container finished" podID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerID="5206beac1eea3f61a7c05bfb4b79220c63b1bc4c832bce9c96f23b1ad8d4191b" exitCode=0 Nov 03 18:48:06.319293 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:06.318954 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerDied","Data":"5206beac1eea3f61a7c05bfb4b79220c63b1bc4c832bce9c96f23b1ad8d4191b"} Nov 03 18:48:07.441494 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.441472 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:48:07.560052 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.560024 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\") pod \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " Nov 03 18:48:07.560224 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.560135 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\" (UniqueName: \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\") pod \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " Nov 03 18:48:07.560224 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.560205 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\") pod \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\" (UID: \"209c4af3-fbd2-4f97-b7cc-fcfd63222186\") " Nov 03 18:48:07.560466 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.560446 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle" pod "209c4af3-fbd2-4f97-b7cc-fcfd63222186" (UID: "209c4af3-fbd2-4f97-b7cc-fcfd63222186"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:48:07.562617 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.562584 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl" pod "209c4af3-fbd2-4f97-b7cc-fcfd63222186" (UID: "209c4af3-fbd2-4f97-b7cc-fcfd63222186"). InnerVolumeSpecName "kube-api-access-8j7pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:48:07.566029 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.566001 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util" pod "209c4af3-fbd2-4f97-b7cc-fcfd63222186" (UID: "209c4af3-fbd2-4f97-b7cc-fcfd63222186"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:48:07.660922 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.660886 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:07.660922 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.660916 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/209c4af3-fbd2-4f97-b7cc-fcfd63222186-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:07.660922 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:07.660926 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7pl\" (UniqueName: \"kubernetes.io/projected/209c4af3-fbd2-4f97-b7cc-fcfd63222186-kube-api-access-8j7pl\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:08.327535 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:08.327499 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" event={"ID":"209c4af3-fbd2-4f97-b7cc-fcfd63222186","Type":"ContainerDied","Data":"ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831"} Nov 03 18:48:08.327702 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:08.327547 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb8d3931e394dc7fb0c6d8f83613d287e02c0c1e6c5737e7a8a3982b69af831" Nov 03 18:48:08.327702 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:08.327509 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85djfvh" Nov 03 18:48:09.309989 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:09.309958 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" Nov 03 18:48:10.770964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.770933 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs"] Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771236 5209 cpu_manager.go:414] "containerMap: removing container" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="pull" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771248 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="pull" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771261 5209 cpu_manager.go:414] "containerMap: removing container" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="util" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771267 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="util" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771273 5209 cpu_manager.go:414] "containerMap: removing container" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="extract" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771278 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="extract" Nov 03 18:48:10.771349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.771338 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="209c4af3-fbd2-4f97-b7cc-fcfd63222186" containerName="extract" Nov 03 18:48:10.774371 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.774352 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.783558 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.783538 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn5m\" (UniqueName: \"kubernetes.io/projected/fce07264-908b-409d-be49-7cc597bb70f2-kube-api-access-jvn5m\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.783658 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.783584 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fce07264-908b-409d-be49-7cc597bb70f2-tmp\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.869964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.869936 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:48:10.869964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.869949 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Nov 03 18:48:10.869964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.869936 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-lsdzk\"" Nov 03 18:48:10.870798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.870782 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs"] Nov 03 18:48:10.884263 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.884244 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn5m\" (UniqueName: \"kubernetes.io/projected/fce07264-908b-409d-be49-7cc597bb70f2-kube-api-access-jvn5m\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.884362 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.884283 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fce07264-908b-409d-be49-7cc597bb70f2-tmp\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.884607 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.884593 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fce07264-908b-409d-be49-7cc597bb70f2-tmp\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:10.942964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:10.942930 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn5m\" (UniqueName: \"kubernetes.io/projected/fce07264-908b-409d-be49-7cc597bb70f2-kube-api-access-jvn5m\") pod \"openshift-lws-operator-8465994787-jzwfs\" (UID: \"fce07264-908b-409d-be49-7cc597bb70f2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:11.085630 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:11.085606 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" Nov 03 18:48:11.448489 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:11.448416 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs"] Nov 03 18:48:11.455845 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:48:11.455811 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfce07264_908b_409d_be49_7cc597bb70f2.slice/crio-86e7b4f78d11326ea43efea8072edd7e03a2634788f4b04bb344aed233aeff8b WatchSource:0}: Error finding container 86e7b4f78d11326ea43efea8072edd7e03a2634788f4b04bb344aed233aeff8b: Status 404 returned error can't find the container with id 86e7b4f78d11326ea43efea8072edd7e03a2634788f4b04bb344aed233aeff8b Nov 03 18:48:12.343472 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:12.343433 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" event={"ID":"fce07264-908b-409d-be49-7cc597bb70f2","Type":"ContainerStarted","Data":"86e7b4f78d11326ea43efea8072edd7e03a2634788f4b04bb344aed233aeff8b"} Nov 03 18:48:16.360864 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:16.360828 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" event={"ID":"fce07264-908b-409d-be49-7cc597bb70f2","Type":"ContainerStarted","Data":"7b80b8ba8643b8321cbab2d08b6fc28961ab781e1ecb6d4b0ff8c0bcb16f4b54"} Nov 03 18:48:16.469304 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:16.469256 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jzwfs" podStartSLOduration=2.352444279 podStartE2EDuration="6.469244188s" totalImagesPullingTime="4.116799909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:16.46916634 +0000 UTC m=+177.880266460" watchObservedRunningTime="2026-11-03 18:48:16.469244188 +0000 UTC m=+177.880344310" Nov 03 18:48:19.316602 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.316571 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp"] Nov 03 18:48:19.319076 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.319061 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.346322 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.346289 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.346458 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.346328 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmst\" (UniqueName: \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.346497 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.346467 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.415038 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.414997 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 18:48:19.415038 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.415001 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 18:48:19.415038 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.415042 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xnr87\"" Nov 03 18:48:19.416168 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.416146 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp"] Nov 03 18:48:19.447619 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.447590 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.447766 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.447629 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.447766 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.447656 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmst\" (UniqueName: \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.447976 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.447960 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.448018 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.447997 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.505978 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.505946 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmst\" (UniqueName: \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.631648 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.631618 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:19.995856 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:19.995780 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp"] Nov 03 18:48:20.002592 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:48:20.002567 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86e4c21_c860_4782_96d8_0a0385fb53fb.slice/crio-8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541 WatchSource:0}: Error finding container 8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541: Status 404 returned error can't find the container with id 8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541 Nov 03 18:48:20.377362 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:20.377324 5209 generic.go:356] "Generic (PLEG): container finished" podID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerID="4daebacf84f9a50d34273badee9a75eefd5e9d067e0e6ca0e193e778acf8e2f8" exitCode=0 Nov 03 18:48:20.377362 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:20.377365 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" event={"ID":"b86e4c21-c860-4782-96d8-0a0385fb53fb","Type":"ContainerDied","Data":"4daebacf84f9a50d34273badee9a75eefd5e9d067e0e6ca0e193e778acf8e2f8"} Nov 03 18:48:20.377896 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:20.377400 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" event={"ID":"b86e4c21-c860-4782-96d8-0a0385fb53fb","Type":"ContainerStarted","Data":"8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541"} Nov 03 18:48:22.387360 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:22.387323 5209 generic.go:356] "Generic (PLEG): container finished" podID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerID="2b21bde06875c516181059b8302a660c7460c46d32032db0c7496fb2ec6e6b61" exitCode=0 Nov 03 18:48:22.387763 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:22.387366 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" event={"ID":"b86e4c21-c860-4782-96d8-0a0385fb53fb","Type":"ContainerDied","Data":"2b21bde06875c516181059b8302a660c7460c46d32032db0c7496fb2ec6e6b61"} Nov 03 18:48:23.393138 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:23.393081 5209 generic.go:356] "Generic (PLEG): container finished" podID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerID="dbc16c9dcbcf5c8633cb412308419d1c424c1246eef9917d15ffda8f1b73de5b" exitCode=0 Nov 03 18:48:23.393579 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:23.393172 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" event={"ID":"b86e4c21-c860-4782-96d8-0a0385fb53fb","Type":"ContainerDied","Data":"dbc16c9dcbcf5c8633cb412308419d1c424c1246eef9917d15ffda8f1b73de5b"} Nov 03 18:48:24.512442 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.512420 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:24.585165 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.585137 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\" (UniqueName: \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\") pod \"b86e4c21-c860-4782-96d8-0a0385fb53fb\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " Nov 03 18:48:24.585269 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.585224 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\") pod \"b86e4c21-c860-4782-96d8-0a0385fb53fb\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " Nov 03 18:48:24.585269 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.585249 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\") pod \"b86e4c21-c860-4782-96d8-0a0385fb53fb\" (UID: \"b86e4c21-c860-4782-96d8-0a0385fb53fb\") " Nov 03 18:48:24.585771 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.585747 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle" pod "b86e4c21-c860-4782-96d8-0a0385fb53fb" (UID: "b86e4c21-c860-4782-96d8-0a0385fb53fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:48:24.587131 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.587094 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst" pod "b86e4c21-c860-4782-96d8-0a0385fb53fb" (UID: "b86e4c21-c860-4782-96d8-0a0385fb53fb"). InnerVolumeSpecName "kube-api-access-nvmst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:48:24.590402 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.590378 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util" pod "b86e4c21-c860-4782-96d8-0a0385fb53fb" (UID: "b86e4c21-c860-4782-96d8-0a0385fb53fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:48:24.686370 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.686288 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:24.686370 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.686317 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86e4c21-c860-4782-96d8-0a0385fb53fb-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:24.686370 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:24.686331 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmst\" (UniqueName: \"kubernetes.io/projected/b86e4c21-c860-4782-96d8-0a0385fb53fb-kube-api-access-nvmst\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:25.402485 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:25.402457 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" Nov 03 18:48:25.402485 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:25.402474 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714wlwp" event={"ID":"b86e4c21-c860-4782-96d8-0a0385fb53fb","Type":"ContainerDied","Data":"8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541"} Nov 03 18:48:25.402684 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:25.402515 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d999c7b64ead06c1de1962b13abb46d1b75d3cdfbf8eb73239464f5bb30e541" Nov 03 18:48:31.935583 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:48:31.935503 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:48:34.717186 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717154 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s"] Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717427 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="extract" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717438 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="extract" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717454 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="pull" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717459 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="pull" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717468 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="util" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717472 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="util" Nov 03 18:48:34.717554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.717517 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86e4c21-c860-4782-96d8-0a0385fb53fb" containerName="extract" Nov 03 18:48:34.719582 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.719567 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763635 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763606 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6xd\" (UniqueName: \"kubernetes.io/projected/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-kube-api-access-dp6xd\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763644 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763671 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763735 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763965 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763798 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.763965 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.763875 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.768847 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.768828 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 18:48:34.815330 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.815284 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 18:48:34.815491 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.815473 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 18:48:34.815534 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.815493 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tqcfl\"" Nov 03 18:48:34.815534 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.815473 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Nov 03 18:48:34.815534 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.815497 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 18:48:34.835092 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.835065 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s"] Nov 03 18:48:34.864550 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864522 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6xd\" (UniqueName: \"kubernetes.io/projected/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-kube-api-access-dp6xd\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.864659 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864557 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.864659 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864582 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.864741 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864691 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.864789 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864766 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.864849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.864807 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.865143 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.865095 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.866864 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.866845 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.867217 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.867188 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.867498 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.867482 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.867561 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.867543 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:34.939630 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:34.939597 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6xd\" (UniqueName: \"kubernetes.io/projected/aaceaf09-bcca-45f5-8b4d-2923f64e66e0-kube-api-access-dp6xd\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pdm9s\" (UID: \"aaceaf09-bcca-45f5-8b4d-2923f64e66e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:35.032065 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:35.031990 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:35.516738 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:35.516695 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s"] Nov 03 18:48:35.523808 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:48:35.523772 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaceaf09_bcca_45f5_8b4d_2923f64e66e0.slice/crio-bce37bbd334b232c279f978049d2103c6c4dec783841886cc93ec1847960d17d WatchSource:0}: Error finding container bce37bbd334b232c279f978049d2103c6c4dec783841886cc93ec1847960d17d: Status 404 returned error can't find the container with id bce37bbd334b232c279f978049d2103c6c4dec783841886cc93ec1847960d17d Nov 03 18:48:36.449335 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:36.449292 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" event={"ID":"aaceaf09-bcca-45f5-8b4d-2923f64e66e0","Type":"ContainerStarted","Data":"bce37bbd334b232c279f978049d2103c6c4dec783841886cc93ec1847960d17d"} Nov 03 18:48:39.464792 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:39.464706 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" event={"ID":"aaceaf09-bcca-45f5-8b4d-2923f64e66e0","Type":"ContainerStarted","Data":"aadc9cdc6987a4ed6b4649f5933908ab4450f964f839ffc0578baa7bb2d3f0a3"} Nov 03 18:48:40.470774 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:40.470735 5209 generic.go:356] "Generic (PLEG): container finished" podID="aaceaf09-bcca-45f5-8b4d-2923f64e66e0" containerID="aadc9cdc6987a4ed6b4649f5933908ab4450f964f839ffc0578baa7bb2d3f0a3" exitCode=0 Nov 03 18:48:40.471171 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:40.470775 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" event={"ID":"aaceaf09-bcca-45f5-8b4d-2923f64e66e0","Type":"ContainerDied","Data":"aadc9cdc6987a4ed6b4649f5933908ab4450f964f839ffc0578baa7bb2d3f0a3"} Nov 03 18:48:41.476614 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:41.476572 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" event={"ID":"aaceaf09-bcca-45f5-8b4d-2923f64e66e0","Type":"ContainerStarted","Data":"5a049555767e5e5f63164f68f7838025884e934400f4a4d0a8784232bb879a39"} Nov 03 18:48:41.477028 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:41.476711 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:41.699390 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:41.699280 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" podStartSLOduration=2.7735906569999997 podStartE2EDuration="7.699266373s" totalImagesPullingTime="4.51005897s" totalInitContainerRuntime="415.616746ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:41.65268238 +0000 UTC m=+203.063782504" watchObservedRunningTime="2026-11-03 18:48:41.699266373 +0000 UTC m=+203.110366492" Nov 03 18:48:52.484058 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:52.484019 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pdm9s" Nov 03 18:48:53.784183 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.784151 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx"] Nov 03 18:48:53.790738 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.790719 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:53.844336 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.844307 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx"] Nov 03 18:48:53.926572 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.926541 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6576dc50-f730-452b-9e3b-4aaff0527911-manager-config\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:53.926572 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.926577 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zh7\" (UniqueName: \"kubernetes.io/projected/6576dc50-f730-452b-9e3b-4aaff0527911-kube-api-access-c9zh7\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:53.926766 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.926609 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:53.926766 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.926731 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-metrics-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:53.929770 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.929749 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Nov 03 18:48:53.929894 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.929880 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v4rxv\"" Nov 03 18:48:53.930037 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.930022 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Nov 03 18:48:53.930111 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:53.930033 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Nov 03 18:48:54.027324 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.027280 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6576dc50-f730-452b-9e3b-4aaff0527911-manager-config\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.027324 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.027328 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zh7\" (UniqueName: \"kubernetes.io/projected/6576dc50-f730-452b-9e3b-4aaff0527911-kube-api-access-c9zh7\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.027587 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.027359 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.027587 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.027381 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-metrics-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.028006 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.027981 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6576dc50-f730-452b-9e3b-4aaff0527911-manager-config\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.029958 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.029936 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-metrics-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.030052 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.029992 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6576dc50-f730-452b-9e3b-4aaff0527911-cert\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.084151 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.084106 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zh7\" (UniqueName: \"kubernetes.io/projected/6576dc50-f730-452b-9e3b-4aaff0527911-kube-api-access-c9zh7\") pod \"lws-controller-manager-75cc556d68-grxvx\" (UID: \"6576dc50-f730-452b-9e3b-4aaff0527911\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.105909 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.105888 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:54.470542 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.470461 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx"] Nov 03 18:48:54.477021 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:48:54.476996 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6576dc50_f730_452b_9e3b_4aaff0527911.slice/crio-d31f12d5e39920cdc6e55908bda22fad0ef27ce67392d5d599b6eab69416d8d1 WatchSource:0}: Error finding container d31f12d5e39920cdc6e55908bda22fad0ef27ce67392d5d599b6eab69416d8d1: Status 404 returned error can't find the container with id d31f12d5e39920cdc6e55908bda22fad0ef27ce67392d5d599b6eab69416d8d1 Nov 03 18:48:54.533045 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:54.533006 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" event={"ID":"6576dc50-f730-452b-9e3b-4aaff0527911","Type":"ContainerStarted","Data":"d31f12d5e39920cdc6e55908bda22fad0ef27ce67392d5d599b6eab69416d8d1"} Nov 03 18:48:58.550849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:58.550807 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" event={"ID":"6576dc50-f730-452b-9e3b-4aaff0527911","Type":"ContainerStarted","Data":"0ca82ef82f6b334ac5dc57001e76e74f8d44b2486a7d245aba52748e67eb90dd"} Nov 03 18:48:58.551320 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:58.550877 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:48:58.664522 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:48:58.664473 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" podStartSLOduration=2.294801049 podStartE2EDuration="5.664452853s" totalImagesPullingTime="3.369651804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:58.663388412 +0000 UTC m=+220.074488555" watchObservedRunningTime="2026-11-03 18:48:58.664452853 +0000 UTC m=+220.075552975" Nov 03 18:49:01.937600 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:01.937557 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:49:09.557795 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:09.557764 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-grxvx" Nov 03 18:49:10.463507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.463470 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78"] Nov 03 18:49:10.465582 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.465562 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466036 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466004 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466194 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466055 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466248 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466208 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466298 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466268 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvg45\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-kube-api-access-nvg45\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466375 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466354 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466496 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466385 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4ed226-0f3b-429e-bda5-2348b4719de4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466496 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466412 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466496 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466458 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.466674 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.466505 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.533767 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.533711 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b82hw\"" Nov 03 18:49:10.533929 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.533909 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:49:10.533970 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.533909 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 18:49:10.563235 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.563205 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:49:10.564387 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.564365 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78"] Nov 03 18:49:10.566861 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566831 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.566960 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566876 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvg45\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-kube-api-access-nvg45\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.566960 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566908 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.566960 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566933 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4ed226-0f3b-429e-bda5-2348b4719de4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.566960 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566951 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567192 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566972 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567192 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.566991 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567192 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567019 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567192 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567043 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567484 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567463 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567550 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567506 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567610 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567572 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567816 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567793 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.567974 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.567950 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4ed226-0f3b-429e-bda5-2348b4719de4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.570048 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.570020 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.570154 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.570042 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.624235 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.624206 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.624388 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.624271 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvg45\" (UniqueName: \"kubernetes.io/projected/ec4ed226-0f3b-429e-bda5-2348b4719de4-kube-api-access-nvg45\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-rpj78\" (UID: \"ec4ed226-0f3b-429e-bda5-2348b4719de4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:10.777371 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:10.777300 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:11.194941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.194907 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78"] Nov 03 18:49:11.201295 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:49:11.201259 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4ed226_0f3b_429e_bda5_2348b4719de4.slice/crio-003d81238ae977f4320255b1bdbf9e2ddcd19f30554f5e210f9f4d881e0ad86c WatchSource:0}: Error finding container 003d81238ae977f4320255b1bdbf9e2ddcd19f30554f5e210f9f4d881e0ad86c: Status 404 returned error can't find the container with id 003d81238ae977f4320255b1bdbf9e2ddcd19f30554f5e210f9f4d881e0ad86c Nov 03 18:49:11.388054 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.388019 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-r8rjd"] Nov 03 18:49:11.391136 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.391105 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:11.439993 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.439967 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Nov 03 18:49:11.440247 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.440232 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-qhcnh\"" Nov 03 18:49:11.446909 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.446860 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-r8rjd"] Nov 03 18:49:11.474984 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.474963 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:11.475099 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.474999 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8jd\" (UniqueName: \"kubernetes.io/projected/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-kube-api-access-rr8jd\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:11.576036 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.576000 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8jd\" (UniqueName: \"kubernetes.io/projected/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-kube-api-access-rr8jd\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:11.576529 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.576146 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:11.576529 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:11.576276 5209 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Nov 03 18:49:11.576529 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:11.576356 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs podName:1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e nodeName:}" failed. No retries permitted until 2026-11-03 18:49:12.076338916 +0000 UTC m=+233.487439021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs") pod "odh-observability-584f99c788-r8rjd" (UID: "1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e") : secret "odh-observability-webhook-cert" not found Nov 03 18:49:11.604052 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.604020 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" event={"ID":"ec4ed226-0f3b-429e-bda5-2348b4719de4","Type":"ContainerStarted","Data":"003d81238ae977f4320255b1bdbf9e2ddcd19f30554f5e210f9f4d881e0ad86c"} Nov 03 18:49:11.633728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:11.633702 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8jd\" (UniqueName: \"kubernetes.io/projected/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-kube-api-access-rr8jd\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:12.079364 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:12.079331 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:12.079536 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:12.079488 5209 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Nov 03 18:49:12.079577 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:12.079551 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs podName:1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e nodeName:}" failed. No retries permitted until 2026-11-03 18:49:13.079537361 +0000 UTC m=+234.490637475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs") pod "odh-observability-584f99c788-r8rjd" (UID: "1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e") : secret "odh-observability-webhook-cert" not found Nov 03 18:49:13.086978 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:13.086943 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:13.089348 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:13.089322 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e-webhook-certs\") pod \"odh-observability-584f99c788-r8rjd\" (UID: \"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e\") " pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:13.203177 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:13.203150 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:13.581914 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:13.581867 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-r8rjd"] Nov 03 18:49:13.593270 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:49:13.593242 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8bdf57_5c5e_4bdf_84f5_5b58b700db5e.slice/crio-1e2034895aab90a0445274538d84557e87b1dfa9d4020150bcbd750402d8b8a3 WatchSource:0}: Error finding container 1e2034895aab90a0445274538d84557e87b1dfa9d4020150bcbd750402d8b8a3: Status 404 returned error can't find the container with id 1e2034895aab90a0445274538d84557e87b1dfa9d4020150bcbd750402d8b8a3 Nov 03 18:49:13.614840 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:13.614811 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-r8rjd" event={"ID":"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e","Type":"ContainerStarted","Data":"1e2034895aab90a0445274538d84557e87b1dfa9d4020150bcbd750402d8b8a3"} Nov 03 18:49:17.518083 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.518050 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:17.518461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.518150 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:17.518461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.518200 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:17.639016 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.638979 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" event={"ID":"ec4ed226-0f3b-429e-bda5-2348b4719de4","Type":"ContainerStarted","Data":"6010110e7376180f688fa1fab1507c6d6b0537299d7834b7745d807e8e6fef93"} Nov 03 18:49:17.759014 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.758960 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" podStartSLOduration=1.444849297 podStartE2EDuration="7.758940231s" totalImagesPullingTime="6.314090934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:17.757801904 +0000 UTC m=+239.168902042" watchObservedRunningTime="2026-11-03 18:49:17.758940231 +0000 UTC m=+239.170040355" Nov 03 18:49:17.777698 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:17.777639 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:18.644277 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:18.644237 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-r8rjd" event={"ID":"1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e","Type":"ContainerStarted","Data":"9bc1a911dfe4dc8f9142524eaaf4f84fcd4d676bbb06dd69ef77dce2bd512265"} Nov 03 18:49:18.644681 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:18.644469 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:18.754687 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:18.754638 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-r8rjd" podStartSLOduration=3.651846466 podStartE2EDuration="7.754626324s" totalImagesPullingTime="4.102779858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:18.753892871 +0000 UTC m=+240.164992987" watchObservedRunningTime="2026-11-03 18:49:18.754626324 +0000 UTC m=+240.165726426" Nov 03 18:49:18.783411 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:18.783382 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:19.648961 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:19.648927 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:19.649899 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:19.649878 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-rpj78" Nov 03 18:49:26.953080 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:26.953051 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn"] Nov 03 18:49:26.957798 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:26.957779 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.009660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.009632 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e3dc000-f9fd-43e0-b572-c14082077892-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.009793 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.009685 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.009793 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.009750 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlgj\" (UniqueName: \"kubernetes.io/projected/5e3dc000-f9fd-43e0-b572-c14082077892-kube-api-access-xtlgj\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.009872 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.009804 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.110888 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.110861 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e3dc000-f9fd-43e0-b572-c14082077892-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.111082 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.110917 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.111082 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.110942 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlgj\" (UniqueName: \"kubernetes.io/projected/5e3dc000-f9fd-43e0-b572-c14082077892-kube-api-access-xtlgj\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.111082 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.110998 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.111304 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.111286 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.111349 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.111329 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e3dc000-f9fd-43e0-b572-c14082077892-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.147661 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.147628 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-j2jkn\"" Nov 03 18:49:27.147826 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.147663 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 18:49:27.148955 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.148931 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn"] Nov 03 18:49:27.151806 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.151789 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e3dc000-f9fd-43e0-b572-c14082077892-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.169854 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.169836 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlgj\" (UniqueName: \"kubernetes.io/projected/5e3dc000-f9fd-43e0-b572-c14082077892-kube-api-access-xtlgj\") pod \"kserve-module-controller-manager-5cfb77bf8c-2sbkn\" (UID: \"5e3dc000-f9fd-43e0-b572-c14082077892\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.269720 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.269647 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:27.686755 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:27.686723 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn"] Nov 03 18:49:27.693337 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:49:27.693312 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3dc000_f9fd_43e0_b572_c14082077892.slice/crio-0e226fcda35e5890723b2a6e31d1bde24da859863bbadd44e9a3faf68692d258 WatchSource:0}: Error finding container 0e226fcda35e5890723b2a6e31d1bde24da859863bbadd44e9a3faf68692d258: Status 404 returned error can't find the container with id 0e226fcda35e5890723b2a6e31d1bde24da859863bbadd44e9a3faf68692d258 Nov 03 18:49:28.689176 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:28.689139 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" event={"ID":"5e3dc000-f9fd-43e0-b572-c14082077892","Type":"ContainerStarted","Data":"0e226fcda35e5890723b2a6e31d1bde24da859863bbadd44e9a3faf68692d258"} Nov 03 18:49:29.651171 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:29.651112 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-r8rjd" Nov 03 18:49:30.699803 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:30.699764 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" event={"ID":"5e3dc000-f9fd-43e0-b572-c14082077892","Type":"ContainerStarted","Data":"5ea2f3e3a9898f0bfe412dac9ccf5fa4f01623a63b2223d22e3034a3555d330a"} Nov 03 18:49:30.700216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:30.699929 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:49:30.860171 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:30.860102 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" podStartSLOduration=2.082305673 podStartE2EDuration="4.860086874s" totalImagesPullingTime="2.777781201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:30.859178417 +0000 UTC m=+252.270278560" watchObservedRunningTime="2026-11-03 18:49:30.860086874 +0000 UTC m=+252.271186997" Nov 03 18:49:31.935942 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:49:31.935905 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:49:51.705516 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:49:51.705486 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-2sbkn" Nov 03 18:50:01.936658 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:50:01.936577 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:50:25.058572 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.058537 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xsfxn"] Nov 03 18:50:25.066948 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.066919 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.117748 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.117716 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xsfxn"] Nov 03 18:50:25.120884 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.120855 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:50:25.123383 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.123365 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.162988 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.162957 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dzd5p\"" Nov 03 18:50:25.163141 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.163035 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 18:50:25.181650 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.181618 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:50:25.219441 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219407 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 18:50:25.219603 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219489 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5bbrz\"" Nov 03 18:50:25.219709 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219690 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qz22\" (UniqueName: \"kubernetes.io/projected/8ede7475-dec7-4897-adc6-53c013df7362-kube-api-access-5qz22\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.219777 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219764 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjjc\" (UniqueName: \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.219811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219788 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.219811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.219807 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ede7475-dec7-4897-adc6-53c013df7362-cert\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.320843 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.320768 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qz22\" (UniqueName: \"kubernetes.io/projected/8ede7475-dec7-4897-adc6-53c013df7362-kube-api-access-5qz22\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.320843 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.320841 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjjc\" (UniqueName: \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.321063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.320861 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.321063 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.320881 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ede7475-dec7-4897-adc6-53c013df7362-cert\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.323364 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.323343 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ede7475-dec7-4897-adc6-53c013df7362-cert\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.323450 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.323433 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.379412 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.379383 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjjc\" (UniqueName: \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\") pod \"llmisvc-controller-manager-869d5dcc5d-qwrfw\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.379555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.379383 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qz22\" (UniqueName: \"kubernetes.io/projected/8ede7475-dec7-4897-adc6-53c013df7362-kube-api-access-5qz22\") pod \"kserve-controller-manager-776cc66d47-xsfxn\" (UID: \"8ede7475-dec7-4897-adc6-53c013df7362\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.379555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.379505 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:25.438528 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.438490 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:25.744034 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.744002 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xsfxn"] Nov 03 18:50:25.751011 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:50:25.750979 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ede7475_dec7_4897_adc6_53c013df7362.slice/crio-7f41d287bfb3ea522948f783d139e2770840f75052addd8369385cc5c3a1c64b WatchSource:0}: Error finding container 7f41d287bfb3ea522948f783d139e2770840f75052addd8369385cc5c3a1c64b: Status 404 returned error can't find the container with id 7f41d287bfb3ea522948f783d139e2770840f75052addd8369385cc5c3a1c64b Nov 03 18:50:25.805767 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.805742 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:50:25.812434 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:50:25.812402 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4798d76_22c6_4552_882f_5abee668bdf4.slice/crio-4c4fe561881518d616b346cf08a4a8afe51bb23e56cf962a5513c9e5a0b3f07f WatchSource:0}: Error finding container 4c4fe561881518d616b346cf08a4a8afe51bb23e56cf962a5513c9e5a0b3f07f: Status 404 returned error can't find the container with id 4c4fe561881518d616b346cf08a4a8afe51bb23e56cf962a5513c9e5a0b3f07f Nov 03 18:50:25.921545 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.921508 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" event={"ID":"8ede7475-dec7-4897-adc6-53c013df7362","Type":"ContainerStarted","Data":"7f41d287bfb3ea522948f783d139e2770840f75052addd8369385cc5c3a1c64b"} Nov 03 18:50:25.922678 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:25.922654 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" event={"ID":"a4798d76-22c6-4552-882f-5abee668bdf4","Type":"ContainerStarted","Data":"4c4fe561881518d616b346cf08a4a8afe51bb23e56cf962a5513c9e5a0b3f07f"} Nov 03 18:50:29.941975 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:29.941939 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" event={"ID":"a4798d76-22c6-4552-882f-5abee668bdf4","Type":"ContainerStarted","Data":"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d"} Nov 03 18:50:29.942493 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:29.942079 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:50:30.052044 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:30.051979 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" podStartSLOduration=1.529558233 podStartE2EDuration="5.051963564s" totalImagesPullingTime="3.522405331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:30.051052537 +0000 UTC m=+311.462152740" watchObservedRunningTime="2026-11-03 18:50:30.051963564 +0000 UTC m=+311.463063687" Nov 03 18:50:30.948313 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:30.948262 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" event={"ID":"8ede7475-dec7-4897-adc6-53c013df7362","Type":"ContainerStarted","Data":"8367824b8d51108e62f7c8fc4e427a6d6f8a9b01ed8694cde3b14fa615583dbf"} Nov 03 18:50:30.948849 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:30.948433 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:50:31.056734 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:50:31.056682 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" podStartSLOduration=2.615645617 podStartE2EDuration="7.056670275s" totalImagesPullingTime="4.441024658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:31.05652045 +0000 UTC m=+312.467620592" watchObservedRunningTime="2026-11-03 18:50:31.056670275 +0000 UTC m=+312.467770398" Nov 03 18:50:31.936089 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:50:31.936048 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:51:00.950440 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:00.950410 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:51:01.935767 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:51:01.935732 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:51:01.957735 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:01.957708 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xsfxn" Nov 03 18:51:31.935766 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:51:31.935678 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:51:36.142708 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.142674 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r"] Nov 03 18:51:36.145136 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.145101 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.194234 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.194206 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 18:51:36.194343 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.194260 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 18:51:36.240863 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.240838 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5xk7m\"" Nov 03 18:51:36.242321 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.242302 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r"] Nov 03 18:51:36.243343 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.243326 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx"] Nov 03 18:51:36.245602 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.245584 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.294760 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.294731 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-9q568\"" Nov 03 18:51:36.307441 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.307415 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx"] Nov 03 18:51:36.313592 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.313564 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbh5z\" (UniqueName: \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.313692 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.313613 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.313751 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.313734 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.337343 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.337314 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f"] Nov 03 18:51:36.340243 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.340226 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.395139 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.395065 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-64jl5\"" Nov 03 18:51:36.397450 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.397425 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f"] Nov 03 18:51:36.414363 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414339 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.414464 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414382 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbh5z\" (UniqueName: \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.414464 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414405 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.414544 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414463 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xfwt\" (UniqueName: \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.414587 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414543 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.414587 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414576 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.414762 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414747 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.414837 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.414822 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.440730 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.440705 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5"] Nov 03 18:51:36.459055 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.459028 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.473460 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.473435 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbh5z\" (UniqueName: \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.500499 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.500471 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5"] Nov 03 18:51:36.507952 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.507931 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-k7cvz\"" Nov 03 18:51:36.515497 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515374 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.515497 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515446 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfwt\" (UniqueName: \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.515497 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515492 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.515696 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515521 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.515696 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515553 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.515696 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515569 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82ck\" (UniqueName: \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.515941 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.515923 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.516025 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.516004 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.573054 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.573021 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfwt\" (UniqueName: \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.616083 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616054 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.616242 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616091 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.616242 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616108 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82ck\" (UniqueName: \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.616242 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616148 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.616405 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616371 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zht2t\" (UniqueName: \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.616554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616534 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.616597 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616556 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.616748 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.616734 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.674429 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.674367 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82ck\" (UniqueName: \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:36.717193 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.717157 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.717329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.717211 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.717329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.717262 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zht2t\" (UniqueName: \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.717603 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.717572 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.717603 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.717575 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.757237 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.757204 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:36.775492 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.775458 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zht2t\" (UniqueName: \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:36.858185 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.858153 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:36.952561 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:36.952500 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:37.071748 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.071717 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:37.128861 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.128830 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r"] Nov 03 18:51:37.134372 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:51:37.134346 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3300111_3e1d_4ccc_9516_405b2f0325c0.slice/crio-bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829 WatchSource:0}: Error finding container bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829: Status 404 returned error can't find the container with id bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829 Nov 03 18:51:37.227305 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.227278 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx"] Nov 03 18:51:37.244239 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.244209 5209 generic.go:356] "Generic (PLEG): container finished" podID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerID="5f7b33ce9005ca17c1f04e757ad9557beb49516d22b195798f197eadca8524e7" exitCode=0 Nov 03 18:51:37.244373 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.244354 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" event={"ID":"f3300111-3e1d-4ccc-9516-405b2f0325c0","Type":"ContainerDied","Data":"5f7b33ce9005ca17c1f04e757ad9557beb49516d22b195798f197eadca8524e7"} Nov 03 18:51:37.244438 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.244409 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" event={"ID":"f3300111-3e1d-4ccc-9516-405b2f0325c0","Type":"ContainerStarted","Data":"bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829"} Nov 03 18:51:37.245544 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.245526 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:51:37.260824 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:51:37.260794 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc794c1aa_e6fa_4f4a_9aac_7c11fe0e8453.slice/crio-c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006 WatchSource:0}: Error finding container c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006: Status 404 returned error can't find the container with id c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006 Nov 03 18:51:37.323095 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.323069 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f"] Nov 03 18:51:37.329019 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:51:37.328993 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e02980_2981_4d39_afbd_7a1dc64f6628.slice/crio-7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9 WatchSource:0}: Error finding container 7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9: Status 404 returned error can't find the container with id 7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9 Nov 03 18:51:37.436176 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:37.436150 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5"] Nov 03 18:51:37.442515 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:51:37.442491 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7047ea5_4e62_46ff_ae4d_59434c70ed80.slice/crio-28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a WatchSource:0}: Error finding container 28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a: Status 404 returned error can't find the container with id 28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a Nov 03 18:51:38.250559 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.250523 5209 generic.go:356] "Generic (PLEG): container finished" podID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerID="86e8cbce96c8293db6003a9a8092ed36aa870625a07b90d44637db6cca594eb0" exitCode=0 Nov 03 18:51:38.250949 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.250610 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" event={"ID":"e7047ea5-4e62-46ff-ae4d-59434c70ed80","Type":"ContainerDied","Data":"86e8cbce96c8293db6003a9a8092ed36aa870625a07b90d44637db6cca594eb0"} Nov 03 18:51:38.250949 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.250663 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" event={"ID":"e7047ea5-4e62-46ff-ae4d-59434c70ed80","Type":"ContainerStarted","Data":"28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a"} Nov 03 18:51:38.252234 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.252206 5209 generic.go:356] "Generic (PLEG): container finished" podID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerID="d9157f2ad3706d9e28c121f8fba939a7ecb3841a40c9d4ba14668ec5d3a89c88" exitCode=0 Nov 03 18:51:38.252345 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.252308 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" event={"ID":"c2e02980-2981-4d39-afbd-7a1dc64f6628","Type":"ContainerDied","Data":"d9157f2ad3706d9e28c121f8fba939a7ecb3841a40c9d4ba14668ec5d3a89c88"} Nov 03 18:51:38.252412 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.252356 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" event={"ID":"c2e02980-2981-4d39-afbd-7a1dc64f6628","Type":"ContainerStarted","Data":"7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9"} Nov 03 18:51:38.254195 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.254061 5209 generic.go:356] "Generic (PLEG): container finished" podID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerID="d6e683d7522bfdcd72febf91f2c344c7e4a0dd9b67a55e16a31a37b462a01846" exitCode=0 Nov 03 18:51:38.254195 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.254094 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerDied","Data":"d6e683d7522bfdcd72febf91f2c344c7e4a0dd9b67a55e16a31a37b462a01846"} Nov 03 18:51:38.254195 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:38.254152 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerStarted","Data":"c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006"} Nov 03 18:51:39.266332 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:39.266298 5209 generic.go:356] "Generic (PLEG): container finished" podID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerID="530c4c0a58b5005c21e97f27e84db37a3f419249998ed33343c2cc0e04a6d504" exitCode=0 Nov 03 18:51:39.266658 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:39.266376 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" event={"ID":"f3300111-3e1d-4ccc-9516-405b2f0325c0","Type":"ContainerDied","Data":"530c4c0a58b5005c21e97f27e84db37a3f419249998ed33343c2cc0e04a6d504"} Nov 03 18:51:40.272756 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:40.272669 5209 generic.go:356] "Generic (PLEG): container finished" podID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerID="c36d16db4b507d23b51869b1616cbd0b5cb8a65c33752e2c0ba54bc12b88ee67" exitCode=0 Nov 03 18:51:40.273149 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:40.272766 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" event={"ID":"f3300111-3e1d-4ccc-9516-405b2f0325c0","Type":"ContainerDied","Data":"c36d16db4b507d23b51869b1616cbd0b5cb8a65c33752e2c0ba54bc12b88ee67"} Nov 03 18:51:41.399187 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.399161 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:41.457400 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.457375 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\") pod \"f3300111-3e1d-4ccc-9516-405b2f0325c0\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " Nov 03 18:51:41.457571 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.457412 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\") pod \"f3300111-3e1d-4ccc-9516-405b2f0325c0\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " Nov 03 18:51:41.458067 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.458033 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle" pod "f3300111-3e1d-4ccc-9516-405b2f0325c0" (UID: "f3300111-3e1d-4ccc-9516-405b2f0325c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:41.462640 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.462621 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util" pod "f3300111-3e1d-4ccc-9516-405b2f0325c0" (UID: "f3300111-3e1d-4ccc-9516-405b2f0325c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:41.558526 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.558447 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\" (UniqueName: \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\") pod \"f3300111-3e1d-4ccc-9516-405b2f0325c0\" (UID: \"f3300111-3e1d-4ccc-9516-405b2f0325c0\") " Nov 03 18:51:41.558670 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.558609 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:41.558670 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.558625 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3300111-3e1d-4ccc-9516-405b2f0325c0-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:41.560468 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.560445 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z" pod "f3300111-3e1d-4ccc-9516-405b2f0325c0" (UID: "f3300111-3e1d-4ccc-9516-405b2f0325c0"). InnerVolumeSpecName "kube-api-access-xbh5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:51:41.659098 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:41.659066 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbh5z\" (UniqueName: \"kubernetes.io/projected/f3300111-3e1d-4ccc-9516-405b2f0325c0-kube-api-access-xbh5z\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:42.283757 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:42.283730 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" Nov 03 18:51:42.283757 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:42.283745 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anhn5r" event={"ID":"f3300111-3e1d-4ccc-9516-405b2f0325c0","Type":"ContainerDied","Data":"bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829"} Nov 03 18:51:42.283964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:42.283788 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc178749c3ddb585b9b2cc8842f85de10ff75429bdbaba82df1f1fa75fde1829" Nov 03 18:51:44.294127 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:44.294093 5209 generic.go:356] "Generic (PLEG): container finished" podID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerID="b46a8aaaa76c6fae4dbc6dfcc24533fc9e871263208ded0b44a75c9a266502e9" exitCode=0 Nov 03 18:51:44.294484 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:44.294178 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" event={"ID":"e7047ea5-4e62-46ff-ae4d-59434c70ed80","Type":"ContainerDied","Data":"b46a8aaaa76c6fae4dbc6dfcc24533fc9e871263208ded0b44a75c9a266502e9"} Nov 03 18:51:45.303109 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:45.303071 5209 generic.go:356] "Generic (PLEG): container finished" podID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerID="bd1377fd58cc62b7c03b30b4c849c630370ddc74689ca42817919835b5148ab6" exitCode=0 Nov 03 18:51:45.303518 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:45.303146 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" event={"ID":"e7047ea5-4e62-46ff-ae4d-59434c70ed80","Type":"ContainerDied","Data":"bd1377fd58cc62b7c03b30b4c849c630370ddc74689ca42817919835b5148ab6"} Nov 03 18:51:46.431948 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.431924 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:46.498272 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.498241 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\") pod \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " Nov 03 18:51:46.498409 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.498298 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\" (UniqueName: \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\") pod \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " Nov 03 18:51:46.498409 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.498344 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\") pod \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\" (UID: \"e7047ea5-4e62-46ff-ae4d-59434c70ed80\") " Nov 03 18:51:46.498890 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.498867 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle" pod "e7047ea5-4e62-46ff-ae4d-59434c70ed80" (UID: "e7047ea5-4e62-46ff-ae4d-59434c70ed80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:46.500294 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.500277 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t" pod "e7047ea5-4e62-46ff-ae4d-59434c70ed80" (UID: "e7047ea5-4e62-46ff-ae4d-59434c70ed80"). InnerVolumeSpecName "kube-api-access-zht2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:51:46.504437 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.504410 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util" pod "e7047ea5-4e62-46ff-ae4d-59434c70ed80" (UID: "e7047ea5-4e62-46ff-ae4d-59434c70ed80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:46.599311 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.599276 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:46.599311 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.599301 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zht2t\" (UniqueName: \"kubernetes.io/projected/e7047ea5-4e62-46ff-ae4d-59434c70ed80-kube-api-access-zht2t\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:46.599311 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:46.599314 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7047ea5-4e62-46ff-ae4d-59434c70ed80-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:47.314029 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:47.313993 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" Nov 03 18:51:47.314208 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:47.313998 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czbhz5" event={"ID":"e7047ea5-4e62-46ff-ae4d-59434c70ed80","Type":"ContainerDied","Data":"28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a"} Nov 03 18:51:47.314208 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:47.314129 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28cfaadbd9eeeca09a69d4d4e956716624574f58ff94a1559b3d780cb7f0e60a" Nov 03 18:51:48.321490 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:48.321455 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerStarted","Data":"050467ff370ba6222239a27e9a18e430dbc9e6127019ff8f7d88735932cfbd44"} Nov 03 18:51:49.328513 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:49.328482 5209 generic.go:356] "Generic (PLEG): container finished" podID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerID="050467ff370ba6222239a27e9a18e430dbc9e6127019ff8f7d88735932cfbd44" exitCode=0 Nov 03 18:51:49.328920 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:49.328562 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerDied","Data":"050467ff370ba6222239a27e9a18e430dbc9e6127019ff8f7d88735932cfbd44"} Nov 03 18:51:50.335555 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:50.335516 5209 generic.go:356] "Generic (PLEG): container finished" podID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerID="b153b45d30c2107b659824bc47e9399ef697f76d438d75c2f63bff4273ad416a" exitCode=0 Nov 03 18:51:50.335932 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:50.335596 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerDied","Data":"b153b45d30c2107b659824bc47e9399ef697f76d438d75c2f63bff4273ad416a"} Nov 03 18:51:51.462838 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.462812 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:51.535728 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.535700 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\" (UniqueName: \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\") pod \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " Nov 03 18:51:51.535844 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.535808 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\") pod \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " Nov 03 18:51:51.535920 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.535851 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\") pod \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\" (UID: \"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453\") " Nov 03 18:51:51.536287 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.536263 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle" pod "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" (UID: "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:51.537758 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.537736 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt" pod "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" (UID: "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453"). InnerVolumeSpecName "kube-api-access-9xfwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:51:51.540024 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.539997 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util" pod "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" (UID: "c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:51.637325 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.637245 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xfwt\" (UniqueName: \"kubernetes.io/projected/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-kube-api-access-9xfwt\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:51.637325 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.637273 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:51.637325 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:51.637283 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:52.346018 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:52.345979 5209 generic.go:356] "Generic (PLEG): container finished" podID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerID="60835f8115cd6f6b7591464447b343fa82bc6ea235fafe8b18a558ef3ed163b3" exitCode=0 Nov 03 18:51:52.346167 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:52.346057 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" event={"ID":"c2e02980-2981-4d39-afbd-7a1dc64f6628","Type":"ContainerDied","Data":"60835f8115cd6f6b7591464447b343fa82bc6ea235fafe8b18a558ef3ed163b3"} Nov 03 18:51:52.348591 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:52.348567 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" event={"ID":"c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453","Type":"ContainerDied","Data":"c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006"} Nov 03 18:51:52.348687 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:52.348616 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c50e76f87251f1319c3d4dc67cc3705a9259eeaf404d339316c5afb8bde8c006" Nov 03 18:51:52.348687 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:52.348618 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rxcgx" Nov 03 18:51:53.355526 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:53.355493 5209 generic.go:356] "Generic (PLEG): container finished" podID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerID="615964b5b89270404016d6e58552c6d550ff7ef4b7646405a762ae149e8aa4c8" exitCode=0 Nov 03 18:51:53.355885 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:53.355580 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" event={"ID":"c2e02980-2981-4d39-afbd-7a1dc64f6628","Type":"ContainerDied","Data":"615964b5b89270404016d6e58552c6d550ff7ef4b7646405a762ae149e8aa4c8"} Nov 03 18:51:54.478485 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.478463 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:54.558266 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.558235 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\") pod \"c2e02980-2981-4d39-afbd-7a1dc64f6628\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " Nov 03 18:51:54.558414 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.558295 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\") pod \"c2e02980-2981-4d39-afbd-7a1dc64f6628\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " Nov 03 18:51:54.558414 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.558335 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\" (UniqueName: \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\") pod \"c2e02980-2981-4d39-afbd-7a1dc64f6628\" (UID: \"c2e02980-2981-4d39-afbd-7a1dc64f6628\") " Nov 03 18:51:54.558809 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.558783 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle" pod "c2e02980-2981-4d39-afbd-7a1dc64f6628" (UID: "c2e02980-2981-4d39-afbd-7a1dc64f6628"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:54.560462 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.560435 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck" pod "c2e02980-2981-4d39-afbd-7a1dc64f6628" (UID: "c2e02980-2981-4d39-afbd-7a1dc64f6628"). InnerVolumeSpecName "kube-api-access-x82ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:51:54.562507 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.562484 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util" pod "c2e02980-2981-4d39-afbd-7a1dc64f6628" (UID: "c2e02980-2981-4d39-afbd-7a1dc64f6628"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:54.660017 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.659932 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82ck\" (UniqueName: \"kubernetes.io/projected/c2e02980-2981-4d39-afbd-7a1dc64f6628-kube-api-access-x82ck\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:54.660017 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.659961 5209 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-bundle\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:54.660017 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:54.659970 5209 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2e02980-2981-4d39-afbd-7a1dc64f6628-util\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:55.366497 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:55.366468 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" Nov 03 18:51:55.366642 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:55.366464 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vq8f" event={"ID":"c2e02980-2981-4d39-afbd-7a1dc64f6628","Type":"ContainerDied","Data":"7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9"} Nov 03 18:51:55.366642 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:51:55.366599 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5a6bbca367a3eedd71d45ad7d79e83c902a2e964dbd50a0f37547e02322bd9" Nov 03 18:52:01.936586 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:01.936545 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:52:23.744524 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744490 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg"] Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744832 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744842 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744854 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744859 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744866 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744871 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744891 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744896 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744906 5209 cpu_manager.go:414] "containerMap: removing container" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744911 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744922 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744927 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744933 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744937 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744943 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744948 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="extract" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744957 5209 cpu_manager.go:414] "containerMap: removing container" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744961 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744969 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744973 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="util" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744981 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744987 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744994 5209 cpu_manager.go:414] "containerMap: removing container" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="pull" Nov 03 18:52:23.744986 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.744998 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="pull" Nov 03 18:52:23.745901 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.745047 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7047ea5-4e62-46ff-ae4d-59434c70ed80" containerName="extract" Nov 03 18:52:23.745901 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.745055 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e02980-2981-4d39-afbd-7a1dc64f6628" containerName="extract" Nov 03 18:52:23.745901 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.745061 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c794c1aa-e6fa-4f4a-9aac-7c11fe0e8453" containerName="extract" Nov 03 18:52:23.745901 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.745068 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3300111-3e1d-4ccc-9516-405b2f0325c0" containerName="extract" Nov 03 18:52:23.749702 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.749682 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.802873 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.802843 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg"] Nov 03 18:52:23.810446 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.810421 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.810573 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.810556 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tv4w\" (UniqueName: \"kubernetes.io/projected/a1eb1eee-95b6-46aa-b05c-d37803323bf2-kube-api-access-5tv4w\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.810621 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.810596 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1eb1eee-95b6-46aa-b05c-d37803323bf2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.847410 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.847387 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 18:52:23.847410 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.847387 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 18:52:23.847554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.847387 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 18:52:23.852631 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.852613 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 18:52:23.852780 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.852764 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfk6d\"" Nov 03 18:52:23.911530 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.911502 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tv4w\" (UniqueName: \"kubernetes.io/projected/a1eb1eee-95b6-46aa-b05c-d37803323bf2-kube-api-access-5tv4w\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.911698 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.911541 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1eb1eee-95b6-46aa-b05c-d37803323bf2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.911698 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.911569 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.911698 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:23.911679 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:23.911797 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:23.911738 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:24.411721959 +0000 UTC m=+425.822822060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:23.912233 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.912214 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1eb1eee-95b6-46aa-b05c-d37803323bf2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:23.971424 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:23.971398 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tv4w\" (UniqueName: \"kubernetes.io/projected/a1eb1eee-95b6-46aa-b05c-d37803323bf2-kube-api-access-5tv4w\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:24.415858 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:24.415824 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:24.416034 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:24.415947 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:24.416034 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:24.416005 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:25.415989439 +0000 UTC m=+426.827089565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:25.426486 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:25.426450 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:25.426886 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:25.426610 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:25.426886 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:25.426686 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:27.426669695 +0000 UTC m=+428.837769795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:27.441411 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:27.441361 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:27.441857 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:27.441514 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:27.441857 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:27.441581 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:31.441566269 +0000 UTC m=+432.852666382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:31.478521 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:31.478478 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:31.479015 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:31.478655 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:31.479015 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:31.478747 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:39.478725051 +0000 UTC m=+440.889825166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:31.937108 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:31.937070 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:52:39.546760 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:39.546728 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:39.547202 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:39.546896 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:39.547202 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:39.546963 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:52:55.546948455 +0000 UTC m=+456.958048559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:52:55.589776 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:52:55.589734 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:52:55.590333 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:55.589880 5209 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 18:52:55.590333 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:52:55.589956 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert podName:a1eb1eee-95b6-46aa-b05c-d37803323bf2 nodeName:}" failed. No retries permitted until 2026-11-03 18:53:27.589940241 +0000 UTC m=+489.001040346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-nlmfg" (UID: "a1eb1eee-95b6-46aa-b05c-d37803323bf2") : secret "plugin-serving-cert" not found Nov 03 18:53:01.936645 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:53:01.936560 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:53:24.191131 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.191083 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 18:53:24.194372 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.194357 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.244417 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.244383 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-26cvs\"" Nov 03 18:53:24.292722 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.292688 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 18:53:24.338931 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.338901 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339095 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.338964 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339095 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339020 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339095 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339063 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6d2m\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339243 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339111 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339243 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339160 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339306 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339263 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339306 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339301 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.339411 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.339322 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440370 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440334 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440541 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440382 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440541 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440401 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440541 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440516 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440707 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440605 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440707 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440687 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440716 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440746 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6d2m\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440768 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440788 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.440811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440772 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.441032 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.440830 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.441032 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.441017 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.441519 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.441473 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.442856 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.442829 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.443216 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.443194 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.497831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.497799 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.498174 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.498157 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6d2m\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\") pod \"maas-default-gateway-openshift-default-77f556d647-hlqmb\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.507006 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.506986 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:24.929900 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.929864 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 18:53:24.935611 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:53:24.935577 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5972fb_ca3d_404f_96a2_39f40041cb28.slice/crio-0e97b5e725bbd90bdd65b87bed3a459dddc4dd7145eb174d8648fe22b538cc1c WatchSource:0}: Error finding container 0e97b5e725bbd90bdd65b87bed3a459dddc4dd7145eb174d8648fe22b538cc1c: Status 404 returned error can't find the container with id 0e97b5e725bbd90bdd65b87bed3a459dddc4dd7145eb174d8648fe22b538cc1c Nov 03 18:53:24.937521 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.937494 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:53:24.937597 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.937558 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:53:24.937631 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:24.937601 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:53:25.805196 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:25.805155 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerStarted","Data":"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7"} Nov 03 18:53:25.805603 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:25.805221 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerStarted","Data":"0e97b5e725bbd90bdd65b87bed3a459dddc4dd7145eb174d8648fe22b538cc1c"} Nov 03 18:53:25.967519 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:25.967467 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podStartSLOduration=1.967454602 podStartE2EDuration="1.967454602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:25.965659056 +0000 UTC m=+487.376759209" watchObservedRunningTime="2026-11-03 18:53:25.967454602 +0000 UTC m=+487.378554725" Nov 03 18:53:26.507534 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:26.507504 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:26.512288 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:26.512265 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:26.809803 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:26.809722 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:26.810597 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:26.810580 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:53:27.669601 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:27.669570 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:53:27.671868 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:27.671844 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb1eee-95b6-46aa-b05c-d37803323bf2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nlmfg\" (UID: \"a1eb1eee-95b6-46aa-b05c-d37803323bf2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:53:28.008662 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:28.008587 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfk6d\"" Nov 03 18:53:28.011672 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:28.011644 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" Nov 03 18:53:28.378352 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:28.378319 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg"] Nov 03 18:53:28.384147 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:53:28.384090 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1eb1eee_95b6_46aa_b05c_d37803323bf2.slice/crio-29c6e981e81241a0b818fdd554e02c47fd5c6abafc1011b32d3e7ea89e86740c WatchSource:0}: Error finding container 29c6e981e81241a0b818fdd554e02c47fd5c6abafc1011b32d3e7ea89e86740c: Status 404 returned error can't find the container with id 29c6e981e81241a0b818fdd554e02c47fd5c6abafc1011b32d3e7ea89e86740c Nov 03 18:53:28.819556 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:28.819469 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" event={"ID":"a1eb1eee-95b6-46aa-b05c-d37803323bf2","Type":"ContainerStarted","Data":"29c6e981e81241a0b818fdd554e02c47fd5c6abafc1011b32d3e7ea89e86740c"} Nov 03 18:53:31.935637 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:53:31.935595 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:53:33.967325 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:33.967291 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:53:33.970819 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:33.970802 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.019689 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.019660 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 18:53:34.067761 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.067731 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:53:34.124881 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.124839 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpkm\" (UniqueName: \"kubernetes.io/projected/1788a2f1-9a31-481c-8fea-2f79d39a37a0-kube-api-access-drpkm\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.124881 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.124888 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1788a2f1-9a31-481c-8fea-2f79d39a37a0-config-file\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.225706 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.225632 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpkm\" (UniqueName: \"kubernetes.io/projected/1788a2f1-9a31-481c-8fea-2f79d39a37a0-kube-api-access-drpkm\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.225706 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.225672 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1788a2f1-9a31-481c-8fea-2f79d39a37a0-config-file\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.226254 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.226237 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1788a2f1-9a31-481c-8fea-2f79d39a37a0-config-file\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.282993 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.282962 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpkm\" (UniqueName: \"kubernetes.io/projected/1788a2f1-9a31-481c-8fea-2f79d39a37a0-kube-api-access-drpkm\") pod \"limitador-limitador-7c9b87d574-9kbss\" (UID: \"1788a2f1-9a31-481c-8fea-2f79d39a37a0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.283785 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.283764 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:34.299699 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.299672 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:53:34.659461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.659432 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:53:34.665478 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:53:34.665450 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1788a2f1_9a31_481c_8fea_2f79d39a37a0.slice/crio-9b0e61bcad0961f0c949b3ae553d46a7a03b547aff13d6be6e716c78dc9a1b50 WatchSource:0}: Error finding container 9b0e61bcad0961f0c949b3ae553d46a7a03b547aff13d6be6e716c78dc9a1b50: Status 404 returned error can't find the container with id 9b0e61bcad0961f0c949b3ae553d46a7a03b547aff13d6be6e716c78dc9a1b50 Nov 03 18:53:34.856802 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:34.856768 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" event={"ID":"1788a2f1-9a31-481c-8fea-2f79d39a37a0","Type":"ContainerStarted","Data":"9b0e61bcad0961f0c949b3ae553d46a7a03b547aff13d6be6e716c78dc9a1b50"} Nov 03 18:53:38.878187 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:38.878153 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" event={"ID":"a1eb1eee-95b6-46aa-b05c-d37803323bf2","Type":"ContainerStarted","Data":"9540f1e631f000f88f80b88308bebc411344ab20c0c9b7c4e3cb4343c391eb2a"} Nov 03 18:53:38.986906 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:38.986853 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nlmfg" podStartSLOduration=66.544895953 podStartE2EDuration="1m15.986840362s" totalImagesPullingTime="9.441944409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:38.985178234 +0000 UTC m=+500.396278371" watchObservedRunningTime="2026-11-03 18:53:38.986840362 +0000 UTC m=+500.397940485" Nov 03 18:53:39.884502 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:39.884465 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" event={"ID":"1788a2f1-9a31-481c-8fea-2f79d39a37a0","Type":"ContainerStarted","Data":"6532aeac38db49920fae32ecd10baca3c3f2ab4ab6834980e0f1142eb3100a1c"} Nov 03 18:53:39.994494 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:39.994437 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" podStartSLOduration=2.839594843 podStartE2EDuration="6.99442056s" totalImagesPullingTime="4.154825717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:39.99335816 +0000 UTC m=+501.404458300" watchObservedRunningTime="2026-11-03 18:53:39.99442056 +0000 UTC m=+501.405520683" Nov 03 18:53:40.889322 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:40.889293 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:53:51.894550 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:53:51.894516 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9kbss" Nov 03 18:54:01.936385 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:54:01.936342 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:54:31.935977 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:54:31.935895 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:55:01.936297 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:55:01.936253 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:55:11.382652 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:11.382615 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:55:19.310020 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.309986 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n"] Nov 03 18:55:19.313596 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.313578 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.351876 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.351844 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrb2\" (UniqueName: \"kubernetes.io/projected/fd0b3242-3191-44dc-b805-14513b03baea-kube-api-access-lqrb2\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.352059 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.351896 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0b3242-3191-44dc-b805-14513b03baea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.368364 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.368335 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n"] Nov 03 18:55:19.452829 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.452791 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrb2\" (UniqueName: \"kubernetes.io/projected/fd0b3242-3191-44dc-b805-14513b03baea-kube-api-access-lqrb2\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.452829 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.452835 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0b3242-3191-44dc-b805-14513b03baea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.455240 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.455218 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0b3242-3191-44dc-b805-14513b03baea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.510111 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.510072 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrb2\" (UniqueName: \"kubernetes.io/projected/fd0b3242-3191-44dc-b805-14513b03baea-kube-api-access-lqrb2\") pod \"llmisvc-controller-manager-6466c554d8-fvh8n\" (UID: \"fd0b3242-3191-44dc-b805-14513b03baea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.626075 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.626031 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:19.991631 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:19.991559 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n"] Nov 03 18:55:19.997598 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:55:19.997569 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0b3242_3191_44dc_b805_14513b03baea.slice/crio-da3fcc558203161d1b57d53b8fffd103e0868af5e69f361a7dae145d6c7b61a4 WatchSource:0}: Error finding container da3fcc558203161d1b57d53b8fffd103e0868af5e69f361a7dae145d6c7b61a4: Status 404 returned error can't find the container with id da3fcc558203161d1b57d53b8fffd103e0868af5e69f361a7dae145d6c7b61a4 Nov 03 18:55:20.353463 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:20.353428 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" event={"ID":"fd0b3242-3191-44dc-b805-14513b03baea","Type":"ContainerStarted","Data":"da3fcc558203161d1b57d53b8fffd103e0868af5e69f361a7dae145d6c7b61a4"} Nov 03 18:55:21.359502 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.359466 5209 generic.go:356] "Generic (PLEG): container finished" podID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerID="12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7" exitCode=0 Nov 03 18:55:21.360005 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.359542 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerDied","Data":"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7"} Nov 03 18:55:21.360091 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.360032 5209 scope.go:122] "RemoveContainer" containerID="12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7" Nov 03 18:55:21.361346 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.361308 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:55:21.361461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.361348 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" event={"ID":"fd0b3242-3191-44dc-b805-14513b03baea","Type":"ContainerStarted","Data":"bf0c0a94a9fe6b4f18129f54ee4c8eab3a17e59bee0eff156872573b302577a1"} Nov 03 18:55:21.361461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.361391 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:55:21.361461 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.361433 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:55:21.361568 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.361490 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:21.647040 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:21.646937 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" podStartSLOduration=1.92563867 podStartE2EDuration="2.646919683s" totalImagesPullingTime="721.281013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:21.645987248 +0000 UTC m=+603.057087392" watchObservedRunningTime="2026-11-03 18:55:21.646919683 +0000 UTC m=+603.058019806" Nov 03 18:55:22.368950 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:22.368912 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerStarted","Data":"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85"} Nov 03 18:55:22.507544 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:22.507506 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:55:22.508541 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:22.508519 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hlqmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 18:55:22.508602 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:22.508578 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 18:55:23.508086 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:23.508044 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hlqmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 18:55:23.508504 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:23.508109 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 18:55:24.507635 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:24.507596 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hlqmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 18:55:24.507806 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:24.507649 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 18:55:25.507948 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:25.507912 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hlqmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 18:55:25.508373 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:25.507978 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 18:55:26.507754 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:26.507716 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hlqmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 18:55:26.507931 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:26.507770 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 18:55:26.810412 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:26.810324 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:55:27.511571 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:27.511545 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:55:27.512532 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:27.512512 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 18:55:31.935851 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:55:31.935818 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:55:52.371064 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.371034 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fvh8n" Nov 03 18:55:52.528761 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.528728 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:55:52.529074 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.529028 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" podUID="a4798d76-22c6-4552-882f-5abee668bdf4" containerName="manager" containerID="cri-o://05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d" gracePeriod=30 Nov 03 18:55:52.779973 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.779952 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:55:52.869422 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.869393 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\" (UniqueName: \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\") pod \"a4798d76-22c6-4552-882f-5abee668bdf4\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " Nov 03 18:55:52.869554 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.869483 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\" (UniqueName: \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\") pod \"a4798d76-22c6-4552-882f-5abee668bdf4\" (UID: \"a4798d76-22c6-4552-882f-5abee668bdf4\") " Nov 03 18:55:52.871446 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.871421 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert" pod "a4798d76-22c6-4552-882f-5abee668bdf4" (UID: "a4798d76-22c6-4552-882f-5abee668bdf4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:55:52.871525 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.871444 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc" pod "a4798d76-22c6-4552-882f-5abee668bdf4" (UID: "a4798d76-22c6-4552-882f-5abee668bdf4"). InnerVolumeSpecName "kube-api-access-swjjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:52.970860 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.970785 5209 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4798d76-22c6-4552-882f-5abee668bdf4-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:52.970860 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:52.970811 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjjc\" (UniqueName: \"kubernetes.io/projected/a4798d76-22c6-4552-882f-5abee668bdf4-kube-api-access-swjjc\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:53.542490 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.542455 5209 generic.go:356] "Generic (PLEG): container finished" podID="a4798d76-22c6-4552-882f-5abee668bdf4" containerID="05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d" exitCode=0 Nov 03 18:55:53.542956 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.542511 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" Nov 03 18:55:53.542956 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.542547 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" event={"ID":"a4798d76-22c6-4552-882f-5abee668bdf4","Type":"ContainerDied","Data":"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d"} Nov 03 18:55:53.542956 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.542647 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw" event={"ID":"a4798d76-22c6-4552-882f-5abee668bdf4","Type":"ContainerDied","Data":"4c4fe561881518d616b346cf08a4a8afe51bb23e56cf962a5513c9e5a0b3f07f"} Nov 03 18:55:53.542956 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.542691 5209 scope.go:122] "RemoveContainer" containerID="05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d" Nov 03 18:55:53.551634 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.551617 5209 scope.go:122] "RemoveContainer" containerID="05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d" Nov 03 18:55:53.551876 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:55:53.551857 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d\": container with ID starting with 05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d not found: ID does not exist" containerID="05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d" Nov 03 18:55:53.551921 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.551885 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d"} err="failed to get container status \"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d\": rpc error: code = NotFound desc = could not find container \"05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d\": container with ID starting with 05d372ba2bac45cd389146d11590920533ea2797b69f550d6f15a2f2b16be50d not found: ID does not exist" Nov 03 18:55:53.698373 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.698340 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:55:53.744778 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:53.744744 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qwrfw"] Nov 03 18:55:54.697608 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:55:54.697575 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4798d76-22c6-4552-882f-5abee668bdf4" path="/var/lib/kubelet/pods/a4798d76-22c6-4552-882f-5abee668bdf4/volumes" Nov 03 18:56:01.936966 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:56:01.936879 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:56:13.521964 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:56:13.521919 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:56:25.918320 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:56:25.918284 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:56:31.936497 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:56:31.936456 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:56:37.499686 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:56:37.499647 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:56:50.082848 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:56:50.082817 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:57:01.937056 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:57:01.937016 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:57:31.936052 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:57:31.935963 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:58:01.936494 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:01.936454 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:58:27.390208 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:27.390169 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:58:28.960330 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:28.960298 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 18:58:28.960714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:28.960647 5209 cpu_manager.go:414] "containerMap: removing container" podUID="a4798d76-22c6-4552-882f-5abee668bdf4" containerName="manager" Nov 03 18:58:28.960714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:28.960657 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4798d76-22c6-4552-882f-5abee668bdf4" containerName="manager" Nov 03 18:58:28.960714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:28.960714 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4798d76-22c6-4552-882f-5abee668bdf4" containerName="manager" Nov 03 18:58:28.963683 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:28.963665 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.012742 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.012711 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-619731\"" Nov 03 18:58:29.032827 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.032796 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 18:58:29.083012 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.082981 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6vzc\" (UniqueName: \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.083169 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.083038 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.084147 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.084110 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-619731-dockercfg-kwndj\"" Nov 03 18:58:29.183737 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.183705 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6vzc\" (UniqueName: \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.183896 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.183757 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.184506 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.184487 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.241179 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.241087 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6vzc\" (UniqueName: \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\") pod \"payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.276112 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.276086 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:29.650765 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.650734 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 18:58:29.656496 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:58:29.656463 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e6185b_2066_4b1f_9259_84ff64e9e28c.slice/crio-56f4db8b839864a2e3e870472c8695c089a9c9c4689f586d22cd3ebb1c55a505 WatchSource:0}: Error finding container 56f4db8b839864a2e3e870472c8695c089a9c9c4689f586d22cd3ebb1c55a505: Status 404 returned error can't find the container with id 56f4db8b839864a2e3e870472c8695c089a9c9c4689f586d22cd3ebb1c55a505 Nov 03 18:58:29.658303 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:29.658287 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:58:30.285251 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:30.285215 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" event={"ID":"f2e6185b-2066-4b1f-9259-84ff64e9e28c","Type":"ContainerStarted","Data":"56f4db8b839864a2e3e870472c8695c089a9c9c4689f586d22cd3ebb1c55a505"} Nov 03 18:58:31.938389 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:31.938353 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:58:32.297606 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:32.297566 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" event={"ID":"f2e6185b-2066-4b1f-9259-84ff64e9e28c","Type":"ContainerStarted","Data":"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85"} Nov 03 18:58:32.297774 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:32.297660 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:32.405774 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:32.405718 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" podStartSLOduration=1.910199223 podStartE2EDuration="4.405705106s" totalImagesPullingTime="2.495505883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:32.405244265 +0000 UTC m=+793.816344406" watchObservedRunningTime="2026-11-03 18:58:32.405705106 +0000 UTC m=+793.816805258" Nov 03 18:58:36.125254 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:36.125221 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:58:38.871514 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.871478 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 18:58:38.874790 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.874771 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:38.889775 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.889752 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 18:58:38.893167 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.893102 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:38.923617 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.923592 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-eb6ef8\"" Nov 03 18:58:38.935268 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.935245 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-eb6ef8-dockercfg-khq2k\"" Nov 03 18:58:38.937199 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.937174 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 18:58:38.967971 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.967931 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29hm\" (UniqueName: \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:38.968162 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.967986 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:38.968162 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.968058 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcbh\" (UniqueName: \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:38.968162 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.968092 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:38.968162 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.968112 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:38.968408 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.968240 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:38.971785 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:38.971762 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 18:58:39.023866 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.023827 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:58:39.023866 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.023844 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:58:39.024039 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.023895 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-eb6ef8-serving-cert\"" Nov 03 18:58:39.024039 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.023939 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zdsbl\"" Nov 03 18:58:39.024039 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.024023 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:58:39.069088 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069062 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.069265 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069111 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29hm\" (UniqueName: \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:39.069326 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069307 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.069368 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069347 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcbh\" (UniqueName: \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.069455 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069377 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.069455 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.069409 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:39.069573 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:39.069474 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-serving-cert: secret "maas-api-e2e-worker-w6-eb6ef8-serving-cert" not found Nov 03 18:58:39.069573 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:39.069532 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls podName:06329280-b1fd-478d-89c1-a90a4f85a01a nodeName:}" failed. No retries permitted until 2026-11-03 18:58:39.569510769 +0000 UTC m=+800.980610885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls") pod "maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" (UID: "06329280-b1fd-478d-89c1-a90a4f85a01a") : secret "maas-api-e2e-worker-w6-eb6ef8-serving-cert" not found Nov 03 18:58:39.070081 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.070058 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.070261 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.070244 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:39.071867 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.071843 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.129168 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.129081 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcbh\" (UniqueName: \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.129168 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.129108 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29hm\" (UniqueName: \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\") pod \"payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:39.188253 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.188226 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:39.556734 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.556656 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 18:58:39.562691 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:58:39.562661 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33db903_d784_46d8_af5b_a1b1bbff3a0c.slice/crio-a2143978f29088b90d06ca2f90162d1720af7551f01fe60ae02db880ab6bffe2 WatchSource:0}: Error finding container a2143978f29088b90d06ca2f90162d1720af7551f01fe60ae02db880ab6bffe2: Status 404 returned error can't find the container with id a2143978f29088b90d06ca2f90162d1720af7551f01fe60ae02db880ab6bffe2 Nov 03 18:58:39.573984 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.573956 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.576282 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.576263 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:39.804931 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:39.804903 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:40.177573 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.177539 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 18:58:40.183886 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:58:40.183844 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06329280_b1fd_478d_89c1_a90a4f85a01a.slice/crio-8177ff796c1923e73c3be80fbca6057ac23b376aad76c387409165fb90155fdf WatchSource:0}: Error finding container 8177ff796c1923e73c3be80fbca6057ac23b376aad76c387409165fb90155fdf: Status 404 returned error can't find the container with id 8177ff796c1923e73c3be80fbca6057ac23b376aad76c387409165fb90155fdf Nov 03 18:58:40.339868 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.339833 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" event={"ID":"06329280-b1fd-478d-89c1-a90a4f85a01a","Type":"ContainerStarted","Data":"8177ff796c1923e73c3be80fbca6057ac23b376aad76c387409165fb90155fdf"} Nov 03 18:58:40.341294 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.341266 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" event={"ID":"c33db903-d784-46d8-af5b-a1b1bbff3a0c","Type":"ContainerStarted","Data":"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1"} Nov 03 18:58:40.341419 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.341311 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" event={"ID":"c33db903-d784-46d8-af5b-a1b1bbff3a0c","Type":"ContainerStarted","Data":"a2143978f29088b90d06ca2f90162d1720af7551f01fe60ae02db880ab6bffe2"} Nov 03 18:58:40.341419 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.341333 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:40.451784 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:40.451680 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" podStartSLOduration=2.4516656660000002 podStartE2EDuration="2.451665666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:40.449407788 +0000 UTC m=+801.860507911" watchObservedRunningTime="2026-11-03 18:58:40.451665666 +0000 UTC m=+801.862765830" Nov 03 18:58:43.305795 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:43.305718 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 18:58:43.359393 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:43.359364 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" event={"ID":"06329280-b1fd-478d-89c1-a90a4f85a01a","Type":"ContainerStarted","Data":"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd"} Nov 03 18:58:43.359564 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:43.359416 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:43.525194 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:43.525146 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" podStartSLOduration=2.721700727 podStartE2EDuration="5.525132705s" totalImagesPullingTime="2.803431978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:43.523656155 +0000 UTC m=+804.934756297" watchObservedRunningTime="2026-11-03 18:58:43.525132705 +0000 UTC m=+804.936232818" Nov 03 18:58:46.688990 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:46.688948 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:58:48.663917 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.663888 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 18:58:48.667664 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.667633 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.716868 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.716843 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-90a9bf-serving-cert\"" Nov 03 18:58:48.752399 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.752370 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.752530 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.752417 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kpr\" (UniqueName: \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.752530 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.752511 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.752618 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.752587 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.764864 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.764840 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 18:58:48.853900 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.853867 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.854077 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.853918 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.854077 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.853992 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kpr\" (UniqueName: \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.854077 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.854063 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.854283 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:48.854098 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-serving-cert: secret "maas-api-e2e-worker-w5-90a9bf-serving-cert" not found Nov 03 18:58:48.854283 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:58:48.854233 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls podName:4f7f986b-1ca9-41a2-8756-f9adde07ead0 nodeName:}" failed. No retries permitted until 2026-11-03 18:58:49.354209986 +0000 UTC m=+810.765310105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls") pod "maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" (UID: "4f7f986b-1ca9-41a2-8756-f9adde07ead0") : secret "maas-api-e2e-worker-w5-90a9bf-serving-cert" not found Nov 03 18:58:48.854761 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.854743 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.856362 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.856342 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:48.911841 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:48.911814 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kpr\" (UniqueName: \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:49.358195 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:49.358160 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:49.360530 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:49.360508 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:49.369332 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:49.369311 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 18:58:49.581072 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:49.581045 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:49.950834 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:49.950752 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 18:58:49.956800 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:58:49.956773 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7f986b_1ca9_41a2_8756_f9adde07ead0.slice/crio-0a2d1a6be7bef9046f610e4ecaf2d902c04864e2cb22022fd194db5f036ef046 WatchSource:0}: Error finding container 0a2d1a6be7bef9046f610e4ecaf2d902c04864e2cb22022fd194db5f036ef046: Status 404 returned error can't find the container with id 0a2d1a6be7bef9046f610e4ecaf2d902c04864e2cb22022fd194db5f036ef046 Nov 03 18:58:50.394091 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:50.394055 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" event={"ID":"4f7f986b-1ca9-41a2-8756-f9adde07ead0","Type":"ContainerStarted","Data":"0a2d1a6be7bef9046f610e4ecaf2d902c04864e2cb22022fd194db5f036ef046"} Nov 03 18:58:51.350537 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:51.350506 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 18:58:51.400758 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:51.400719 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" event={"ID":"4f7f986b-1ca9-41a2-8756-f9adde07ead0","Type":"ContainerStarted","Data":"277d495084dbe277aad4b763fe85adb85042e03961af24d46880715b033b7ca3"} Nov 03 18:58:51.400915 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:51.400806 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:51.573097 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:51.573047 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" podStartSLOduration=2.661218844 podStartE2EDuration="3.573033733s" totalImagesPullingTime="911.814889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:51.571953972 +0000 UTC m=+812.983054113" watchObservedRunningTime="2026-11-03 18:58:51.573033733 +0000 UTC m=+812.984133855" Nov 03 18:58:57.411819 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:57.411791 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 18:58:58.049682 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.049645 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 18:58:58.053651 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.053610 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.102598 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.102573 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-92596a-serving-cert\"" Nov 03 18:58:58.109564 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.109536 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 18:58:58.195660 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.195631 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:58:58.237714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.237676 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.237878 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.237735 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.237878 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.237809 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.237878 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.237855 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdb8\" (UniqueName: \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.339160 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.339109 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.339329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.339227 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.339329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.339259 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.339329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.339292 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdb8\" (UniqueName: \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.339863 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.339840 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.341699 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.341680 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.341814 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.341794 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.397572 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.397543 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdb8\" (UniqueName: \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\") pod \"maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:58.665787 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:58.665706 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:58:59.083451 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:59.083420 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 18:58:59.089574 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:58:59.089549 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf5bb02_37fa_48aa_9988_8e55840f1312.slice/crio-ed0f77f58e1389ded3f67501407868a15d7d9c13bfea6871e61401f89bbfa478 WatchSource:0}: Error finding container ed0f77f58e1389ded3f67501407868a15d7d9c13bfea6871e61401f89bbfa478: Status 404 returned error can't find the container with id ed0f77f58e1389ded3f67501407868a15d7d9c13bfea6871e61401f89bbfa478 Nov 03 18:58:59.441177 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:58:59.441090 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" event={"ID":"fdf5bb02-37fa-48aa-9988-8e55840f1312","Type":"ContainerStarted","Data":"ed0f77f58e1389ded3f67501407868a15d7d9c13bfea6871e61401f89bbfa478"} Nov 03 18:59:00.447746 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:00.447708 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" event={"ID":"fdf5bb02-37fa-48aa-9988-8e55840f1312","Type":"ContainerStarted","Data":"0cd0a417f711851d7fe3f4c3400fa77e148b2d7d2bca65ed50b84eeffcc0189a"} Nov 03 18:59:00.448151 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:00.447792 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:59:00.559797 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:00.559743 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" podStartSLOduration=1.633139508 podStartE2EDuration="2.559728624s" totalImagesPullingTime="926.589116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:00.558601525 +0000 UTC m=+821.969701674" watchObservedRunningTime="2026-11-03 18:59:00.559728624 +0000 UTC m=+821.970828747" Nov 03 18:59:01.936868 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:59:01.936829 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:59:03.672921 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.672884 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 18:59:03.676396 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.676378 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683615 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683585 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683725 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683686 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683784 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683726 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4f7\" (UniqueName: \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683784 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683753 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683893 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683875 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.683936 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.683901 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.771961 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.771926 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"kube-root-ca.crt\"" Nov 03 18:59:03.772188 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.772167 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"openshift-service-ca.crt\"" Nov 03 18:59:03.772275 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.772175 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"default-dockercfg-nmqfn\"" Nov 03 18:59:03.772275 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.772235 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"simulator-w4-98860e-kserve-self-signed-certs\"" Nov 03 18:59:03.774112 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.774077 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 18:59:03.784826 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.784800 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4f7\" (UniqueName: \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785136 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785094 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785245 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785189 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785245 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785219 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785365 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785279 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785496 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785458 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785647 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785618 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785748 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785726 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.785811 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.785770 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.787851 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.787831 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.788186 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.788165 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.844831 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.844800 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4f7\" (UniqueName: \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\") pod \"simulator-w4-98860e-kserve-5f4d796567-xjbnr\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:03.987778 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:03.987682 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:04.362224 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:04.362186 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 18:59:04.368223 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:59:04.368192 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883e3469_c996_4d7c_a954_bb81d753e409.slice/crio-90f59f1b714ec5e359ab3411b050ab02bae1b23994adb4475bb62f75dc05a829 WatchSource:0}: Error finding container 90f59f1b714ec5e359ab3411b050ab02bae1b23994adb4475bb62f75dc05a829: Status 404 returned error can't find the container with id 90f59f1b714ec5e359ab3411b050ab02bae1b23994adb4475bb62f75dc05a829 Nov 03 18:59:04.468154 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:04.468103 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" event={"ID":"883e3469-c996-4d7c-a954-bb81d753e409","Type":"ContainerStarted","Data":"90f59f1b714ec5e359ab3411b050ab02bae1b23994adb4475bb62f75dc05a829"} Nov 03 18:59:06.457985 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:06.457955 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 18:59:06.482230 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:06.482194 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" event={"ID":"883e3469-c996-4d7c-a954-bb81d753e409","Type":"ContainerStarted","Data":"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9"} Nov 03 18:59:06.691634 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:06.691565 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podStartSLOduration=1.748356931 podStartE2EDuration="3.691546134s" totalImagesPullingTime="1.943189203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:06.689185608 +0000 UTC m=+828.100285732" watchObservedRunningTime="2026-11-03 18:59:06.691546134 +0000 UTC m=+828.102646258" Nov 03 18:59:06.988420 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:06.988346 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:08.140845 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:08.140812 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:59:13.988696 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:13.988664 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:14.001364 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:14.001341 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:14.536001 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:14.535969 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 18:59:29.150769 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.150740 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 18:59:29.153702 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.153676 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.202971 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.202938 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"default-dockercfg-s6hcs\"" Nov 03 18:59:29.221224 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221189 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.221371 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221263 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2rr\" (UniqueName: \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.221371 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221338 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.221441 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221368 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.221441 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221392 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.221565 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.221544 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.249536 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.249510 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"simulator-w3-fa2878-kserve-self-signed-certs\"" Nov 03 18:59:29.249758 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.249737 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"openshift-service-ca.crt\"" Nov 03 18:59:29.249830 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.249757 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"kube-root-ca.crt\"" Nov 03 18:59:29.255030 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.255002 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 18:59:29.322200 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322165 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2rr\" (UniqueName: \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322392 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322212 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322392 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322239 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322392 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322264 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322392 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322334 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322645 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322421 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322754 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322729 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322812 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322772 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.322812 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.322787 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.324570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.324548 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.325039 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.325022 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.382751 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.382717 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2rr\" (UniqueName: \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\") pod \"simulator-w3-fa2878-kserve-d6fbf4986-tdkd6\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.464257 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.464157 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:29.835623 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:29.835591 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 18:59:29.841654 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:59:29.841627 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d0b444_f833_4f1e_be3a_c7d875ae7799.slice/crio-aa7572bee213ba4c9293496627fa3c658804117b04cbdc2a9bc29c7ffc1b388e WatchSource:0}: Error finding container aa7572bee213ba4c9293496627fa3c658804117b04cbdc2a9bc29c7ffc1b388e: Status 404 returned error can't find the container with id aa7572bee213ba4c9293496627fa3c658804117b04cbdc2a9bc29c7ffc1b388e Nov 03 18:59:30.610724 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:30.610688 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" event={"ID":"43d0b444-f833-4f1e-be3a-c7d875ae7799","Type":"ContainerStarted","Data":"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d"} Nov 03 18:59:30.610724 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:30.610734 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" event={"ID":"43d0b444-f833-4f1e-be3a-c7d875ae7799","Type":"ContainerStarted","Data":"aa7572bee213ba4c9293496627fa3c658804117b04cbdc2a9bc29c7ffc1b388e"} Nov 03 18:59:30.722334 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:30.722221 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podStartSLOduration=1.7222056399999999 podStartE2EDuration="1.72220564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:30.721326431 +0000 UTC m=+852.132426573" watchObservedRunningTime="2026-11-03 18:59:30.72220564 +0000 UTC m=+852.133305765" Nov 03 18:59:31.465239 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:31.465209 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:31.937021 ip-10-0-133-236 kubenswrapper[5209]: E1103 18:59:31.936985 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 18:59:34.636576 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:34.636547 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 18:59:36.997079 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:36.997042 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 18:59:36.999617 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:36.999596 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.048518 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.048495 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"kube-root-ca.crt\"" Nov 03 18:59:37.094959 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.094926 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"openshift-service-ca.crt\"" Nov 03 18:59:37.095082 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.094991 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"default-dockercfg-tk8tf\"" Nov 03 18:59:37.095082 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.095003 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"simulator-w2-619731-kserve-self-signed-certs\"" Nov 03 18:59:37.097240 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.097220 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 18:59:37.098483 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098466 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjstp\" (UniqueName: \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.098570 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098501 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.098630 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098615 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.098692 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098680 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.098726 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098715 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.098761 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.098743 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200295 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200248 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200295 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200294 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200542 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200342 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjstp\" (UniqueName: \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200542 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200390 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200655 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200529 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200749 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200722 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200926 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200753 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.200926 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200792 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.201109 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.200970 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.202822 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.202798 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.203007 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.202991 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.257714 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.257646 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjstp\" (UniqueName: \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\") pod \"simulator-w2-619731-kserve-776559cb6-78prt\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.310836 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.310806 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:37.691179 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:37.691147 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 18:59:37.697415 ip-10-0-133-236 kubenswrapper[5209]: W1103 18:59:37.697390 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaacefc91_48a2_4f34_a016_84d2efbd7256.slice/crio-c9fb70c29b2c54042b36d9e959249c2748c3160d7d4c1ed2c6373663040399fe WatchSource:0}: Error finding container c9fb70c29b2c54042b36d9e959249c2748c3160d7d4c1ed2c6373663040399fe: Status 404 returned error can't find the container with id c9fb70c29b2c54042b36d9e959249c2748c3160d7d4c1ed2c6373663040399fe Nov 03 18:59:38.650998 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:38.650962 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" event={"ID":"aacefc91-48a2-4f34-a016-84d2efbd7256","Type":"ContainerStarted","Data":"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf"} Nov 03 18:59:38.650998 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:38.650996 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" event={"ID":"aacefc91-48a2-4f34-a016-84d2efbd7256","Type":"ContainerStarted","Data":"c9fb70c29b2c54042b36d9e959249c2748c3160d7d4c1ed2c6373663040399fe"} Nov 03 18:59:38.762103 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:38.762046 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podStartSLOduration=2.762032304 podStartE2EDuration="2.762032304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:38.761105512 +0000 UTC m=+860.172205637" watchObservedRunningTime="2026-11-03 18:59:38.762032304 +0000 UTC m=+860.173132426" Nov 03 18:59:39.311329 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:39.311301 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:39.465278 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:39.465241 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:39.477947 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:39.477921 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:39.666725 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:39.666700 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 18:59:47.311706 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:47.311670 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:47.324193 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:47.324161 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:47.708023 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:47.707995 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 18:59:52.173140 ip-10-0-133-236 kubenswrapper[5209]: I1103 18:59:52.173086 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:00:00.178867 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.178828 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:00.181912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.181889 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.202028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.201991 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:00.204331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.204310 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.214822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.214798 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.214911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.214849 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.233165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.233140 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:00:00.281970 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.281941 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:00.283729 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.283701 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:00.316271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.316245 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.316395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.316299 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.316395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.316368 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.316395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.316391 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7fx\" (UniqueName: \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.317033 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.317015 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.374001 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.373967 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\") pod \"maas-api-key-cleanup-29895540-zg24d\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.417501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.417470 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7fx\" (UniqueName: \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.417676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.417530 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.418186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.418167 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.475823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.475767 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7fx\" (UniqueName: \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.494424 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.494401 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:00.515458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.515438 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:00.862553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.862520 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:00.867789 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:00:00.867764 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bcabf6_6a27_474d_8360_24dc492743df.slice/crio-a667e6a2bdf6a9fe46fb111ac5a8ef8add515b650923ff5644fb3b4957c7a65a WatchSource:0}: Error finding container a667e6a2bdf6a9fe46fb111ac5a8ef8add515b650923ff5644fb3b4957c7a65a: Status 404 returned error can't find the container with id a667e6a2bdf6a9fe46fb111ac5a8ef8add515b650923ff5644fb3b4957c7a65a Nov 03 19:00:00.912693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:00.912666 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:00.918831 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:00:00.918803 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dc787f_7179_4faa_976b_0f3c0bc07d01.slice/crio-661168df05eeaf655d4fc4f6c34004a0dfed2b57fe023572df6bbea4e33686cb WatchSource:0}: Error finding container 661168df05eeaf655d4fc4f6c34004a0dfed2b57fe023572df6bbea4e33686cb: Status 404 returned error can't find the container with id 661168df05eeaf655d4fc4f6c34004a0dfed2b57fe023572df6bbea4e33686cb Nov 03 19:00:01.766935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:01.766894 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerStarted","Data":"a667e6a2bdf6a9fe46fb111ac5a8ef8add515b650923ff5644fb3b4957c7a65a"} Nov 03 19:00:01.767993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:01.767966 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" event={"ID":"c5dc787f-7179-4faa-976b-0f3c0bc07d01","Type":"ContainerStarted","Data":"661168df05eeaf655d4fc4f6c34004a0dfed2b57fe023572df6bbea4e33686cb"} Nov 03 19:00:01.937491 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:01.937454 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:00:05.563068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.562983 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:00:05.566446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.566403 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.621590 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.621549 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:00:05.665611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.665576 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"premium-w4-98860e-kserve-self-signed-certs\"" Nov 03 19:00:05.669206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669185 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.669304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669234 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.669304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669258 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.669372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669320 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.669409 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669386 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.669484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.669464 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5w4z\" (UniqueName: \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770587 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770789 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770634 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770789 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770653 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770774 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770911 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.770993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.770961 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5w4z\" (UniqueName: \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.771177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.771041 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.771177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.771076 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.771333 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.771309 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.773073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.773046 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.773367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.773346 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.794664 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.794636 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerStarted","Data":"0c0d99d34435ac023d87bd1c4fff188b6e684e5fa1d5d08753e2e2e50a04ca8e"} Nov 03 19:00:05.829014 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.828987 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5w4z\" (UniqueName: \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\") pod \"premium-w4-98860e-kserve-7f89bc8858-b8dn9\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.877211 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.877170 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:05.901281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:05.901225 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" podStartSLOduration=1.365229373 podStartE2EDuration="5.901210375s" totalImagesPullingTime="4.535981002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:05.900623537 +0000 UTC m=+887.311723662" watchObservedRunningTime="2026-11-03 19:00:05.901210375 +0000 UTC m=+887.312310534" Nov 03 19:00:06.012252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.012201 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" podStartSLOduration=1.8604570969999998 podStartE2EDuration="6.012187058s" totalImagesPullingTime="4.151729961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:06.010427292 +0000 UTC m=+887.421527435" watchObservedRunningTime="2026-11-03 19:00:06.012187058 +0000 UTC m=+887.423287182" Nov 03 19:00:06.262486 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.262450 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:00:06.264814 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:00:06.264788 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7df4092_2d8b_4989_9b44_9a8ef7aa972a.slice/crio-58f704823a14356a617a655820d62832407074f31fb47489457bc69e8a72baa1 WatchSource:0}: Error finding container 58f704823a14356a617a655820d62832407074f31fb47489457bc69e8a72baa1: Status 404 returned error can't find the container with id 58f704823a14356a617a655820d62832407074f31fb47489457bc69e8a72baa1 Nov 03 19:00:06.801158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.801105 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" event={"ID":"d7df4092-2d8b-4989-9b44-9a8ef7aa972a","Type":"ContainerStarted","Data":"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d"} Nov 03 19:00:06.801158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.801163 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" event={"ID":"d7df4092-2d8b-4989-9b44-9a8ef7aa972a","Type":"ContainerStarted","Data":"58f704823a14356a617a655820d62832407074f31fb47489457bc69e8a72baa1"} Nov 03 19:00:06.802680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.802654 5209 generic.go:356] "Generic (PLEG): container finished" podID="f8bcabf6-6a27-474d-8360-24dc492743df" containerID="0c0d99d34435ac023d87bd1c4fff188b6e684e5fa1d5d08753e2e2e50a04ca8e" exitCode=60 Nov 03 19:00:06.802818 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.802731 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerDied","Data":"0c0d99d34435ac023d87bd1c4fff188b6e684e5fa1d5d08753e2e2e50a04ca8e"} Nov 03 19:00:06.802939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.802922 5209 scope.go:122] "RemoveContainer" containerID="0c0d99d34435ac023d87bd1c4fff188b6e684e5fa1d5d08753e2e2e50a04ca8e" Nov 03 19:00:06.804280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.804261 5209 generic.go:356] "Generic (PLEG): container finished" podID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerID="8d674d4785cc8db5cc772bdc692fb0437f96875756c4941feefc16b6c7093858" exitCode=60 Nov 03 19:00:06.804375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.804333 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" event={"ID":"c5dc787f-7179-4faa-976b-0f3c0bc07d01","Type":"ContainerDied","Data":"8d674d4785cc8db5cc772bdc692fb0437f96875756c4941feefc16b6c7093858"} Nov 03 19:00:06.804634 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.804616 5209 scope.go:122] "RemoveContainer" containerID="8d674d4785cc8db5cc772bdc692fb0437f96875756c4941feefc16b6c7093858" Nov 03 19:00:06.878047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.878021 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:06.914859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:06.914811 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podStartSLOduration=1.91479409 podStartE2EDuration="1.91479409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:06.913731438 +0000 UTC m=+888.324831565" watchObservedRunningTime="2026-11-03 19:00:06.91479409 +0000 UTC m=+888.325894213" Nov 03 19:00:07.812840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.812807 5209 generic.go:356] "Generic (PLEG): container finished" podID="f8bcabf6-6a27-474d-8360-24dc492743df" containerID="4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564" exitCode=60 Nov 03 19:00:07.813298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.812889 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerDied","Data":"4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564"} Nov 03 19:00:07.813298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.812970 5209 scope.go:122] "RemoveContainer" containerID="0c0d99d34435ac023d87bd1c4fff188b6e684e5fa1d5d08753e2e2e50a04ca8e" Nov 03 19:00:07.813298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.813216 5209 scope.go:122] "RemoveContainer" containerID="4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564" Nov 03 19:00:07.813487 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:07.813465 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-zg24d_odh-ai-gateway-infra(f8bcabf6-6a27-474d-8360-24dc492743df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" Nov 03 19:00:07.814966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.814942 5209 generic.go:356] "Generic (PLEG): container finished" podID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerID="4c37ff984dd24261bc12b88a7d6e380bb582cc4d22d03a6b61406e33967ce716" exitCode=60 Nov 03 19:00:07.815042 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.815002 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" event={"ID":"c5dc787f-7179-4faa-976b-0f3c0bc07d01","Type":"ContainerDied","Data":"4c37ff984dd24261bc12b88a7d6e380bb582cc4d22d03a6b61406e33967ce716"} Nov 03 19:00:07.815371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.815351 5209 scope.go:122] "RemoveContainer" containerID="4c37ff984dd24261bc12b88a7d6e380bb582cc4d22d03a6b61406e33967ce716" Nov 03 19:00:07.815585 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:07.815567 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq_odh-ai-gateway-infra(c5dc787f-7179-4faa-976b-0f3c0bc07d01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" Nov 03 19:00:07.823684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:07.823666 5209 scope.go:122] "RemoveContainer" containerID="8d674d4785cc8db5cc772bdc692fb0437f96875756c4941feefc16b6c7093858" Nov 03 19:00:08.824358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:08.824325 5209 scope.go:122] "RemoveContainer" containerID="4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564" Nov 03 19:00:08.824811 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:08.824583 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-zg24d_odh-ai-gateway-infra(f8bcabf6-6a27-474d-8360-24dc492743df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" Nov 03 19:00:11.069996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:11.069960 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:00:15.185203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.185169 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:00:15.188664 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.188645 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.237630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.237599 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"kube-root-ca.crt\"" Nov 03 19:00:15.237799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.237630 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"openshift-service-ca.crt\"" Nov 03 19:00:15.261201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261166 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.261362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261250 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.261362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261287 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqj8\" (UniqueName: \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.261362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261349 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.261540 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261374 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.261540 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.261425 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.284346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.284319 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"default-dockercfg-fbmhw\"" Nov 03 19:00:15.284491 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.284385 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"simulator-w5-90a9bf-kserve-self-signed-certs\"" Nov 03 19:00:15.286437 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.286412 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:00:15.362701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362669 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.362870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362748 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.362870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362804 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.362870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362844 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.363025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362886 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqj8\" (UniqueName: \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.363025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.362929 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.363196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.363175 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.363268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.363214 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.363327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.363300 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.365107 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.365084 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.365288 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.365273 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.420828 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.420791 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqj8\" (UniqueName: \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\") pod \"simulator-w5-90a9bf-kserve-648b5d68d9-bj26c\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.501096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.500321 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:15.871682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.871651 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:00:15.877584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.877556 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:15.877991 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:00:15.877964 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5559d644_3b16_4c69_9c9f_58447c6442bc.slice/crio-d4aead6245ffecc91eb1b722bd1334ed1523df66c2b45e88ad6153cf18b9c2d8 WatchSource:0}: Error finding container d4aead6245ffecc91eb1b722bd1334ed1523df66c2b45e88ad6153cf18b9c2d8: Status 404 returned error can't find the container with id d4aead6245ffecc91eb1b722bd1334ed1523df66c2b45e88ad6153cf18b9c2d8 Nov 03 19:00:15.891723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:15.891702 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:16.869328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:16.869283 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" event={"ID":"5559d644-3b16-4c69-9c9f-58447c6442bc","Type":"ContainerStarted","Data":"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871"} Nov 03 19:00:16.869328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:16.869334 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" event={"ID":"5559d644-3b16-4c69-9c9f-58447c6442bc","Type":"ContainerStarted","Data":"d4aead6245ffecc91eb1b722bd1334ed1523df66c2b45e88ad6153cf18b9c2d8"} Nov 03 19:00:16.880650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:16.880623 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:00:16.979585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:16.979532 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podStartSLOduration=1.979518084 podStartE2EDuration="1.979518084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:16.978574903 +0000 UTC m=+898.389675028" watchObservedRunningTime="2026-11-03 19:00:16.979518084 +0000 UTC m=+898.390618208" Nov 03 19:00:17.500600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:17.500568 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:17.692080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:17.692051 5209 scope.go:122] "RemoveContainer" containerID="4c37ff984dd24261bc12b88a7d6e380bb582cc4d22d03a6b61406e33967ce716" Nov 03 19:00:18.882711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:18.882683 5209 generic.go:356] "Generic (PLEG): container finished" podID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerID="f0d3a19bc4823ec066a41f16b916cdd3f9703cbe09dbde43523c0707163ac09d" exitCode=60 Nov 03 19:00:18.883068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:18.882751 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" event={"ID":"c5dc787f-7179-4faa-976b-0f3c0bc07d01","Type":"ContainerDied","Data":"f0d3a19bc4823ec066a41f16b916cdd3f9703cbe09dbde43523c0707163ac09d"} Nov 03 19:00:18.883068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:18.882809 5209 scope.go:122] "RemoveContainer" containerID="4c37ff984dd24261bc12b88a7d6e380bb582cc4d22d03a6b61406e33967ce716" Nov 03 19:00:18.883194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:18.883148 5209 scope.go:122] "RemoveContainer" containerID="f0d3a19bc4823ec066a41f16b916cdd3f9703cbe09dbde43523c0707163ac09d" Nov 03 19:00:18.883417 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:18.883393 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq_odh-ai-gateway-infra(c5dc787f-7179-4faa-976b-0f3c0bc07d01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" Nov 03 19:00:20.000607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.000575 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:20.140993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.140970 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:20.311294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.311220 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\" (UniqueName: \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\") pod \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " Nov 03 19:00:20.311442 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.311312 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\") pod \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\" (UID: \"c5dc787f-7179-4faa-976b-0f3c0bc07d01\") " Nov 03 19:00:20.311660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.311638 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca" pod "c5dc787f-7179-4faa-976b-0f3c0bc07d01" (UID: "c5dc787f-7179-4faa-976b-0f3c0bc07d01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:20.313467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.313446 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx" pod "c5dc787f-7179-4faa-976b-0f3c0bc07d01" (UID: "c5dc787f-7179-4faa-976b-0f3c0bc07d01"). InnerVolumeSpecName "kube-api-access-4d7fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:20.412823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.412788 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dc787f-7179-4faa-976b-0f3c0bc07d01-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:20.412823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.412820 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d7fx\" (UniqueName: \"kubernetes.io/projected/c5dc787f-7179-4faa-976b-0f3c0bc07d01-kube-api-access-4d7fx\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:20.691548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.691443 5209 scope.go:122] "RemoveContainer" containerID="4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564" Nov 03 19:00:20.895936 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.895912 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" Nov 03 19:00:20.896073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.895910 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq" event={"ID":"c5dc787f-7179-4faa-976b-0f3c0bc07d01","Type":"ContainerDied","Data":"661168df05eeaf655d4fc4f6c34004a0dfed2b57fe023572df6bbea4e33686cb"} Nov 03 19:00:20.896073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:20.896036 5209 scope.go:122] "RemoveContainer" containerID="f0d3a19bc4823ec066a41f16b916cdd3f9703cbe09dbde43523c0707163ac09d" Nov 03 19:00:21.053840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.053802 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:21.089005 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.088978 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895540-kg8qq"] Nov 03 19:00:21.903250 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.903215 5209 generic.go:356] "Generic (PLEG): container finished" podID="f8bcabf6-6a27-474d-8360-24dc492743df" containerID="c28f871741a482322cf099a0afeb5e22650b468016a0bdc4cb1fe7f1803de0c9" exitCode=60 Nov 03 19:00:21.903457 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.903285 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerDied","Data":"c28f871741a482322cf099a0afeb5e22650b468016a0bdc4cb1fe7f1803de0c9"} Nov 03 19:00:21.903457 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.903354 5209 scope.go:122] "RemoveContainer" containerID="4ea4d87e4cef9c5605a5dc0d16f900fb8fde425acf39f4a5c41dae2a452ba564" Nov 03 19:00:21.903708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:21.903690 5209 scope.go:122] "RemoveContainer" containerID="c28f871741a482322cf099a0afeb5e22650b468016a0bdc4cb1fe7f1803de0c9" Nov 03 19:00:21.903974 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:21.903945 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-zg24d_odh-ai-gateway-infra(f8bcabf6-6a27-474d-8360-24dc492743df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" Nov 03 19:00:22.697135 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:22.697089 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" path="/var/lib/kubelet/pods/c5dc787f-7179-4faa-976b-0f3c0bc07d01/volumes" Nov 03 19:00:23.020455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.020366 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:23.149459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.149432 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:23.245112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.245084 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\") pod \"f8bcabf6-6a27-474d-8360-24dc492743df\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " Nov 03 19:00:23.245294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.245218 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\") pod \"f8bcabf6-6a27-474d-8360-24dc492743df\" (UID: \"f8bcabf6-6a27-474d-8360-24dc492743df\") " Nov 03 19:00:23.245505 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.245478 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca" pod "f8bcabf6-6a27-474d-8360-24dc492743df" (UID: "f8bcabf6-6a27-474d-8360-24dc492743df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:23.247181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.247159 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f" pod "f8bcabf6-6a27-474d-8360-24dc492743df" (UID: "f8bcabf6-6a27-474d-8360-24dc492743df"). InnerVolumeSpecName "kube-api-access-j4w7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.346849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.346816 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bcabf6-6a27-474d-8360-24dc492743df-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.346849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.346843 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/f8bcabf6-6a27-474d-8360-24dc492743df-kube-api-access-j4w7f\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.917086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.917051 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" event={"ID":"f8bcabf6-6a27-474d-8360-24dc492743df","Type":"ContainerDied","Data":"a667e6a2bdf6a9fe46fb111ac5a8ef8add515b650923ff5644fb3b4957c7a65a"} Nov 03 19:00:23.917535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.917092 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d" Nov 03 19:00:23.917535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:23.917110 5209 scope.go:122] "RemoveContainer" containerID="c28f871741a482322cf099a0afeb5e22650b468016a0bdc4cb1fe7f1803de0c9" Nov 03 19:00:24.070188 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:24.070151 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:24.116815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:24.116782 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-zg24d"] Nov 03 19:00:24.697436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:24.697405 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" path="/var/lib/kubelet/pods/f8bcabf6-6a27-474d-8360-24dc492743df/volumes" Nov 03 19:00:25.501188 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:25.501160 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:25.513286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:25.513260 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:25.940427 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:25.940394 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:00:31.113231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:31.113159 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:00:31.937483 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:00:31.937449 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:00:47.430784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.430737 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:00:47.432037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432001 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432026 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432057 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432065 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432089 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432097 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432131 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432138 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432372 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432383 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432400 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:00:47.432671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432409 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.432671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.432427 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:00:47.439865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.439840 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.489685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.489653 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:00:47.533954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.533922 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"premium-w3-fa2878-kserve-self-signed-certs\"" Nov 03 19:00:47.576441 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576414 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.576599 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576449 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.576599 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576541 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.576689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576622 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbr5\" (UniqueName: \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.576689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576659 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.576779 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.576765 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677760 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677807 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677827 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677868 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677887 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbr5\" (UniqueName: \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.677975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.677907 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.678330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.678297 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.678396 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.678340 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.678458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.678429 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.680726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.680706 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.681113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.681071 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.735650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.735615 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbr5\" (UniqueName: \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\") pod \"premium-w3-fa2878-kserve-77c54769f-mw8bj\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:47.751256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:47.751223 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:48.128581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:48.128548 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:00:48.134488 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:00:48.134463 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776454c0_5160_427e_9ae5_cca43cd496da.slice/crio-aaa42f02eab19c463ca2b1d5d3916802cfe7ecc6dadebdd3343f899d6988b7bd WatchSource:0}: Error finding container aaa42f02eab19c463ca2b1d5d3916802cfe7ecc6dadebdd3343f899d6988b7bd: Status 404 returned error can't find the container with id aaa42f02eab19c463ca2b1d5d3916802cfe7ecc6dadebdd3343f899d6988b7bd Nov 03 19:00:48.177325 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:48.177302 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:00:49.057674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:49.057631 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" event={"ID":"776454c0-5160-427e-9ae5-cca43cd496da","Type":"ContainerStarted","Data":"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f"} Nov 03 19:00:49.058072 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:49.057683 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" event={"ID":"776454c0-5160-427e-9ae5-cca43cd496da","Type":"ContainerStarted","Data":"aaa42f02eab19c463ca2b1d5d3916802cfe7ecc6dadebdd3343f899d6988b7bd"} Nov 03 19:00:49.184338 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:49.184280 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podStartSLOduration=2.184263224 podStartE2EDuration="2.184263224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:49.183452887 +0000 UTC m=+930.594553030" watchObservedRunningTime="2026-11-03 19:00:49.184263224 +0000 UTC m=+930.595363347" Nov 03 19:00:49.751979 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:49.751935 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:57.752194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:57.752159 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:57.764618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:57.764596 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:00:58.127086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:00:58.127061 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:01:01.939742 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:01:01.939698 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:01:08.477241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:08.477207 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:01:31.937259 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:01:31.937223 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:01:42.613873 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:42.613832 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:01:50.163516 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.163476 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:01:50.164154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.164082 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:01:50.164154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.164102 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dc787f-7179-4faa-976b-0f3c0bc07d01" containerName="cleanup" Nov 03 19:01:50.164154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.164137 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:01:50.164154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.164145 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:01:50.164457 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.164261 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bcabf6-6a27-474d-8360-24dc492743df" containerName="cleanup" Nov 03 19:01:50.167802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.167782 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191591 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191628 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191658 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191784 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191890 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcd8g\" (UniqueName: \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.191951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.191923 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.263090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.263063 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"premium-w5-90a9bf-kserve-self-signed-certs\"" Nov 03 19:01:50.265168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.265144 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:01:50.293282 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293257 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293383 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293291 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293383 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293320 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293423 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293484 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcd8g\" (UniqueName: \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293634 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293600 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293746 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293811 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.293959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.293937 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.295495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.295477 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.295839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.295823 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.350796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.350759 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcd8g\" (UniqueName: \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\") pod \"premium-w5-90a9bf-kserve-6bd89b5596-5rwv5\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.479613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.479522 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:50.859162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:50.859111 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:01:50.864380 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:01:50.864353 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89319f39_d4e0_4080_afdb_350f97b3a9bc.slice/crio-6046aad54dc05f675d0d3bbc87d3b5fa945a1ce9bc1a1f482c8be72478774ec8 WatchSource:0}: Error finding container 6046aad54dc05f675d0d3bbc87d3b5fa945a1ce9bc1a1f482c8be72478774ec8: Status 404 returned error can't find the container with id 6046aad54dc05f675d0d3bbc87d3b5fa945a1ce9bc1a1f482c8be72478774ec8 Nov 03 19:01:51.396333 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:51.396303 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" event={"ID":"89319f39-d4e0-4080-afdb-350f97b3a9bc","Type":"ContainerStarted","Data":"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53"} Nov 03 19:01:51.396729 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:51.396341 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" event={"ID":"89319f39-d4e0-4080-afdb-350f97b3a9bc","Type":"ContainerStarted","Data":"6046aad54dc05f675d0d3bbc87d3b5fa945a1ce9bc1a1f482c8be72478774ec8"} Nov 03 19:01:51.479983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:51.479954 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:01:51.506607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:01:51.506560 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podStartSLOduration=1.506546288 podStartE2EDuration="1.506546288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:51.505616099 +0000 UTC m=+992.916716222" watchObservedRunningTime="2026-11-03 19:01:51.506546288 +0000 UTC m=+992.917646410" Nov 03 19:02:00.480532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:00.480443 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:02:00.493290 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:00.493261 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:02:01.463146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:01.463093 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:02:01.937720 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:02:01.937683 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:02:04.502892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.502856 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:02:04.506531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.506511 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.555494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.555465 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"openshift-service-ca.crt\"" Nov 03 19:02:04.602423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.602396 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"kube-root-ca.crt\"" Nov 03 19:02:04.602423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.602410 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"default-dockercfg-pt8rk\"" Nov 03 19:02:04.619741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.619720 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"premium-w0-92596a-kserve-self-signed-certs\"" Nov 03 19:02:04.635391 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635360 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.635510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635438 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.635510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635499 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vffr\" (UniqueName: \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.635600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635544 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.635600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635566 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.635683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.635657 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.652974 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.652932 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:02:04.736476 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736444 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736489 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736524 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736621 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736818 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736674 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736818 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736705 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vffr\" (UniqueName: \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.736975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736908 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.737099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.736995 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.737099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.737068 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.738779 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.738752 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.739024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.739006 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.796347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.796273 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vffr\" (UniqueName: \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\") pod \"premium-w0-92596a-kserve-548bdd4bc8-4s8mf\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:04.818977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:04.818950 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:05.196171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:05.196128 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:02:05.201686 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:02:05.201655 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e8707c1_0c6f_40ad_87ae_39a0b503ca48.slice/crio-dffe7e70df9f8e24a51aaa1add48d2eb4a8c7b775135d0d837cbddb8f06fba2b WatchSource:0}: Error finding container dffe7e70df9f8e24a51aaa1add48d2eb4a8c7b775135d0d837cbddb8f06fba2b: Status 404 returned error can't find the container with id dffe7e70df9f8e24a51aaa1add48d2eb4a8c7b775135d0d837cbddb8f06fba2b Nov 03 19:02:05.475339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:05.475255 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" event={"ID":"5e8707c1-0c6f-40ad-87ae-39a0b503ca48","Type":"ContainerStarted","Data":"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb"} Nov 03 19:02:05.475339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:05.475298 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" event={"ID":"5e8707c1-0c6f-40ad-87ae-39a0b503ca48","Type":"ContainerStarted","Data":"dffe7e70df9f8e24a51aaa1add48d2eb4a8c7b775135d0d837cbddb8f06fba2b"} Nov 03 19:02:05.630716 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:05.630664 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podStartSLOduration=1.630651062 podStartE2EDuration="1.630651062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:05.630435043 +0000 UTC m=+1007.041535167" watchObservedRunningTime="2026-11-03 19:02:05.630651062 +0000 UTC m=+1007.041751186" Nov 03 19:02:05.819604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:05.819533 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:10.837027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:10.836993 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:02:10.841068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:10.841046 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:10.937275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:10.937238 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-98860e-openshift-default-dockercfg-p56jx\"" Nov 03 19:02:10.939708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:10.939686 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:02:11.000343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000312 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000352 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000558 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000690 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000783 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000735 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000783 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000775 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000848 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.000965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.000941 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.001047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.001025 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvm7\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.101980 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvm7\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102041 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102064 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102104 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102159 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102190 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102229 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102255 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102292 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102576 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102604 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102730 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.102943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.102852 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.103194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.103048 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.104477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.104452 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.104878 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.104856 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.159779 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.159744 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvm7\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.159924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.159844 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.454269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.454184 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:11.879613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.879579 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:02:11.887940 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:02:11.887909 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac42e280_2969_42ae_bbc3_68509c342310.slice/crio-23be1e57f48284a98458e8b6ba8ed4ad59db6a2c15d814501e6922e538c2a053 WatchSource:0}: Error finding container 23be1e57f48284a98458e8b6ba8ed4ad59db6a2c15d814501e6922e538c2a053: Status 404 returned error can't find the container with id 23be1e57f48284a98458e8b6ba8ed4ad59db6a2c15d814501e6922e538c2a053 Nov 03 19:02:11.890416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.890384 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:11.890521 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.890466 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:11.890521 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:11.890509 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:12.518550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:12.518514 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" event={"ID":"ac42e280-2969-42ae-bbc3-68509c342310","Type":"ContainerStarted","Data":"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f"} Nov 03 19:02:12.518720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:12.518599 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" event={"ID":"ac42e280-2969-42ae-bbc3-68509c342310","Type":"ContainerStarted","Data":"23be1e57f48284a98458e8b6ba8ed4ad59db6a2c15d814501e6922e538c2a053"} Nov 03 19:02:12.738833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:12.738776 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podStartSLOduration=2.738763561 podStartE2EDuration="2.738763561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:12.690883723 +0000 UTC m=+1014.101983849" watchObservedRunningTime="2026-11-03 19:02:12.738763561 +0000 UTC m=+1014.149863683" Nov 03 19:02:13.455177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:13.455145 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:13.456357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:13.456332 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:13.456478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:13.456390 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:14.455372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:14.455341 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:14.455760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:14.455396 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:14.819427 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:14.819339 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:14.832032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:14.832004 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:15.107951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:15.107920 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:02:15.455254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:15.455166 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:15.455403 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:15.455249 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:15.546571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:15.546541 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:02:16.454515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:16.454482 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:16.454721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:16.454531 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:17.455345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:17.455310 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:17.455761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:17.455373 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:18.454472 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:18.454442 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:02:18.454645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:18.454505 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 19:02:19.539439 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:19.539399 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:19.539920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:19.539725 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:19.540799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:19.540780 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:02:25.625877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.625842 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:02:25.630459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.630442 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.726237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.726201 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fa2878-openshift-default-dockercfg-hfvcf\"" Nov 03 19:02:25.729363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.729329 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:02:25.756905 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.756875 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.756917 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757026 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvpd\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757077 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757100 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757129 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757191 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757280 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.757428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.757305 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.857975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.857950 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.857984 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858005 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858028 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858085 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858180 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858225 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858263 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858346 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvpd\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858606 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858427 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858606 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858474 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858714 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858669 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.858769 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.858742 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.860763 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.859451 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.861001 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.860972 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.861081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.861041 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.915936 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.915866 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.916073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.916025 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvpd\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:25.943256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:25.943231 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:26.365467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.365427 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:02:26.370935 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:02:26.370910 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a8049f_43bb_4c92_b308_e52a6ae6e8a2.slice/crio-071a6f1af8096c3f80e077e9c199c883e7b7b697d8e5499fff79e69becfab392 WatchSource:0}: Error finding container 071a6f1af8096c3f80e077e9c199c883e7b7b697d8e5499fff79e69becfab392: Status 404 returned error can't find the container with id 071a6f1af8096c3f80e077e9c199c883e7b7b697d8e5499fff79e69becfab392 Nov 03 19:02:26.373231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.373198 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:26.373326 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.373263 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:26.373326 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.373289 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:26.596005 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.595954 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" event={"ID":"95a8049f-43bb-4c92-b308-e52a6ae6e8a2","Type":"ContainerStarted","Data":"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19"} Nov 03 19:02:26.596211 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.596031 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" event={"ID":"95a8049f-43bb-4c92-b308-e52a6ae6e8a2","Type":"ContainerStarted","Data":"071a6f1af8096c3f80e077e9c199c883e7b7b697d8e5499fff79e69becfab392"} Nov 03 19:02:26.756186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.756011 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podStartSLOduration=1.755997606 podStartE2EDuration="1.755997606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:26.709307656 +0000 UTC m=+1028.120407812" watchObservedRunningTime="2026-11-03 19:02:26.755997606 +0000 UTC m=+1028.167097729" Nov 03 19:02:26.943493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:26.943466 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:27.944911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:27.944879 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:02:27.945311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:27.944940 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 19:02:28.943750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:28.943721 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:02:28.943934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:28.943770 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 19:02:29.943809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:29.943778 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:02:29.944240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:29.943828 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 19:02:30.944551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:30.944518 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:02:30.944937 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:30.944572 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 19:02:31.937879 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:02:31.937842 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:02:31.944413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:31.944386 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:02:31.944529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:31.944445 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 19:02:33.054697 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:33.054663 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:33.055213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:33.055051 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:33.056053 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:33.056001 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:02:50.729617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:50.729582 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:02:56.790393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.790352 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:02:56.795378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.795355 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.851868 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.851837 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-619731-openshift-default-dockercfg-m7qhx\"" Nov 03 19:02:56.899882 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.899842 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.899891 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.899958 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.899996 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssvd\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.900100 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.900168 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.900318 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.900386 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.900520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.900452 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:56.904078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:56.904053 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:02:57.001481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001448 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001484 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001511 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001534 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssvd\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001572 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001621 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001668 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001703 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.001941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001730 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.002057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.001972 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.002057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.002023 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.002189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.002109 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.002227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.002207 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.002227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.002217 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.003923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.003895 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.004217 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.004201 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.058876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.058809 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssvd\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.058876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.058811 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.108901 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.108874 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:57.534287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.534256 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:02:57.539876 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:02:57.539851 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6ce26c_0fb2_4bc9_b860_196ef1247bc0.slice/crio-d7cbc73255c7b0546312412681ab1ec14ef8781b32bcb2a28e536ecba0adbcd1 WatchSource:0}: Error finding container d7cbc73255c7b0546312412681ab1ec14ef8781b32bcb2a28e536ecba0adbcd1: Status 404 returned error can't find the container with id d7cbc73255c7b0546312412681ab1ec14ef8781b32bcb2a28e536ecba0adbcd1 Nov 03 19:02:57.542032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.542001 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:57.542096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.542065 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:57.542096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.542091 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:57.791085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.791045 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" event={"ID":"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0","Type":"ContainerStarted","Data":"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24"} Nov 03 19:02:57.791509 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.791109 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" event={"ID":"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0","Type":"ContainerStarted","Data":"d7cbc73255c7b0546312412681ab1ec14ef8781b32bcb2a28e536ecba0adbcd1"} Nov 03 19:02:57.951925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:57.951819 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podStartSLOduration=1.951805013 podStartE2EDuration="1.951805013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:57.950841751 +0000 UTC m=+1059.361941875" watchObservedRunningTime="2026-11-03 19:02:57.951805013 +0000 UTC m=+1059.362905136" Nov 03 19:02:58.109336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:58.109300 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:02:58.111041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:58.111016 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:02:58.111176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:58.111062 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:02:59.109271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:59.109227 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:02:59.109660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:02:59.109286 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:03:00.109768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:00.109732 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:03:00.110187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:00.109788 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:03:01.109503 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:01.109465 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:03:01.109683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:01.109523 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:03:01.939093 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:03:01.939053 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:03:02.109640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:02.109604 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:03:02.109796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:02.109662 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:03:03.109802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:03.109763 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:03:03.110252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:03.109826 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 19:03:04.349689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:04.349654 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:03:04.350145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:04.349983 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:03:04.350886 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:04.350866 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:03:11.594143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.594062 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:03:11.598283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.598261 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.647300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.647265 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-90a9bf-openshift-default-dockercfg-jqm4m\"" Nov 03 19:03:11.656288 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656249 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6s8\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656387 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656416 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656642 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656555 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656642 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656631 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656659 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656728 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656840 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.656930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.656868 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.697820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.697773 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:03:11.757812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757778 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.757812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757813 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757886 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757909 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757934 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.757971 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758025 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758067 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758115 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6s8\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758249 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758298 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758569 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.758812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.758788 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.759345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.759320 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.760501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.760472 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.760606 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.760599 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.815927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.815893 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.816065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.816002 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6s8\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:11.911824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:11.911721 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:12.352240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.352138 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:03:12.355993 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:12.355956 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08567cb6_11cc_4575_80c2_001beabfcbae.slice/crio-695598ed758c23a9a865bd12d6a6b0481b7fd8e9dd5111a3bbbb6df1eabd68e2 WatchSource:0}: Error finding container 695598ed758c23a9a865bd12d6a6b0481b7fd8e9dd5111a3bbbb6df1eabd68e2: Status 404 returned error can't find the container with id 695598ed758c23a9a865bd12d6a6b0481b7fd8e9dd5111a3bbbb6df1eabd68e2 Nov 03 19:03:12.359020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.358977 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:12.359150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.359060 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:12.359150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.359104 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:12.630547 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.630460 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:03:12.634944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.634919 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670346 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670423 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670472 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670561 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670640 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670688 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670670 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670689 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670709 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9qh\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.670874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.670852 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.719791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.719762 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-eb6ef8-openshift-default-dockercfg-rxz8d\"" Nov 03 19:03:12.723729 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.723702 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:03:12.772010 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.771978 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772085 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772152 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772224 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772313 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772353 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772593 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772557 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772598 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772623 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772638 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772664 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9qh\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.772811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.772700 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.773071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.773053 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.773176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.773080 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.774592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.774570 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.774782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.774766 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.830392 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.830357 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.830632 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.830533 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9qh\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:12.888041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.887957 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" event={"ID":"08567cb6-11cc-4575-80c2-001beabfcbae","Type":"ContainerStarted","Data":"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a"} Nov 03 19:03:12.888041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.888028 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" event={"ID":"08567cb6-11cc-4575-80c2-001beabfcbae","Type":"ContainerStarted","Data":"695598ed758c23a9a865bd12d6a6b0481b7fd8e9dd5111a3bbbb6df1eabd68e2"} Nov 03 19:03:12.912022 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.911991 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:12.953201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:12.952339 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:13.007231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.007178 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podStartSLOduration=2.007162381 podStartE2EDuration="2.007162381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:13.006465111 +0000 UTC m=+1074.417565258" watchObservedRunningTime="2026-11-03 19:03:13.007162381 +0000 UTC m=+1074.418262504" Nov 03 19:03:13.356052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.356015 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:03:13.362002 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:13.361969 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38aa753f_f902_46e6_9aa6_d64ba51f3472.slice/crio-aca408234c606c6b6c7fe024ba6d504ce4750cd22953fd15efbbda1d1d63a6ce WatchSource:0}: Error finding container aca408234c606c6b6c7fe024ba6d504ce4750cd22953fd15efbbda1d1d63a6ce: Status 404 returned error can't find the container with id aca408234c606c6b6c7fe024ba6d504ce4750cd22953fd15efbbda1d1d63a6ce Nov 03 19:03:13.364380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.364348 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:13.364473 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.364410 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:13.364473 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.364437 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:13.905733 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.905701 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" event={"ID":"38aa753f-f902-46e6-9aa6-d64ba51f3472","Type":"ContainerStarted","Data":"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b"} Nov 03 19:03:13.906353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.906319 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" event={"ID":"38aa753f-f902-46e6-9aa6-d64ba51f3472","Type":"ContainerStarted","Data":"aca408234c606c6b6c7fe024ba6d504ce4750cd22953fd15efbbda1d1d63a6ce"} Nov 03 19:03:13.914509 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.914466 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:13.914639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.914514 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:13.954313 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:13.954280 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:14.114146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.110096 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podStartSLOduration=2.110070288 podStartE2EDuration="2.110070288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:14.069347904 +0000 UTC m=+1075.480448053" watchObservedRunningTime="2026-11-03 19:03:14.110070288 +0000 UTC m=+1075.521170413" Nov 03 19:03:14.209198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.208278 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:03:14.221456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.221426 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.263920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.263852 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:03:14.332143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.318694 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p74n\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.332444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.332416 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.332623 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.332605 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.332743 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.332724 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.332902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.332888 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.333013 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.332999 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.333242 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.333202 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.333371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.333328 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.333436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.333390 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.434802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434761 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434847 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434886 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p74n\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434927 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434954 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.434980 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435046 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435073 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435175 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435276 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435478 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.435600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.435577 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.436104 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.436086 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.436227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.436208 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.437232 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.437210 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.438022 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.438000 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.509185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.508209 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.513141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.510932 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p74n\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\") pod \"maas-default-gateway-openshift-default-77f556d647-z92v4\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.539629 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.539592 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:14.926146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.915610 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:14.926146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.915701 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:14.955436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.955399 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:14.955598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:14.955444 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:15.030650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.030622 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:03:15.030811 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:15.030782 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b4de9e_4778_4fe3_8bd6_b17690da67e6.slice/crio-547b4d7b9c480c95c00368993798302bd8a510b65930fb285a8a0bb7fd529e2b WatchSource:0}: Error finding container 547b4d7b9c480c95c00368993798302bd8a510b65930fb285a8a0bb7fd529e2b: Status 404 returned error can't find the container with id 547b4d7b9c480c95c00368993798302bd8a510b65930fb285a8a0bb7fd529e2b Nov 03 19:03:15.033732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.033691 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:15.033812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.033787 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:15.033847 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.033832 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:15.921141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.920845 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:15.921141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.920908 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:15.957522 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.956831 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:15.957522 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.956921 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:15.967230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.967194 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" event={"ID":"55b4de9e-4778-4fe3-8bd6-b17690da67e6","Type":"ContainerStarted","Data":"6a867e895cd0787d8f32856d9c935b34ddb35e346d0fff6155035b66c31b3228"} Nov 03 19:03:15.967415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:15.967405 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" event={"ID":"55b4de9e-4778-4fe3-8bd6-b17690da67e6","Type":"ContainerStarted","Data":"547b4d7b9c480c95c00368993798302bd8a510b65930fb285a8a0bb7fd529e2b"} Nov 03 19:03:16.144146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.141437 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podStartSLOduration=2.141418805 podStartE2EDuration="2.141418805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:16.088612712 +0000 UTC m=+1077.499712846" watchObservedRunningTime="2026-11-03 19:03:16.141418805 +0000 UTC m=+1077.552518930" Nov 03 19:03:16.547916 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.547878 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:16.548329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.548304 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.548522 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.548496 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:16.913744 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.913302 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.913744 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.913356 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:16.954879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.954848 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.955190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:16.955160 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:17.549145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.548778 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.549145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.548835 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:17.916535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.916495 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.916839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.916805 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:17.955049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.954855 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.955049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:17.954912 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:18.540717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.540681 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.540880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.540743 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:18.912560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.912516 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.913179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.913143 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 19:03:18.954053 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.954016 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.954256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:18.954080 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:19.540809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.540773 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:19.541809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.541767 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:19.916434 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.916405 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:19.916832 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.916811 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:19.917592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.917571 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:03:19.953724 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.953694 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:03:19.953866 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:19.953752 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 19:03:20.540474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:20.540442 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:20.540652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:20.540502 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:20.960745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:20.960716 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:20.961169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:20.960998 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:20.961662 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:20.961642 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:03:21.540834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:21.540799 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:21.541031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:21.540861 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:22.540084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:22.540048 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z92v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:03:22.540592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:22.540130 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 19:03:23.833498 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:23.833458 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:23.833945 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:23.833899 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:23.834875 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:23.834843 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:03:26.412709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.412676 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:03:26.417577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.417556 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.512911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.512876 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:03:26.515904 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.515878 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516019 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.515915 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516065 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2pv\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516102 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516262 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516232 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516356 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516448 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516396 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516519 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516506 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.516557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.516531 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617506 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2pv\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617543 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617669 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617749 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617771 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617853 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617813 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617853 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617842 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617879 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.617932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.617901 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.618088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.618064 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.618249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.618187 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.618317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.618267 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.618317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.618293 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.618497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.618482 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.620047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.620023 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.620322 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.620303 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.675252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.675195 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.675363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.675261 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2pv\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:26.731020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:26.730990 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:27.157536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.157498 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:03:27.162314 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:27.162281 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3906bfb_afdb_4752_907f_695607248522.slice/crio-3479956040f08eb798b1df436258c1e4af63919ee43c89de1bac1b66fc37b15c WatchSource:0}: Error finding container 3479956040f08eb798b1df436258c1e4af63919ee43c89de1bac1b66fc37b15c: Status 404 returned error can't find the container with id 3479956040f08eb798b1df436258c1e4af63919ee43c89de1bac1b66fc37b15c Nov 03 19:03:27.164638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.164604 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:27.164726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.164669 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:27.164726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.164695 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:27.397393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.397363 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:03:27.401110 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.401086 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.426786 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.426758 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.426808 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.426931 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.426969 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.427069 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.427113 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.427178 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.427198 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvskz\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.427426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.427311 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.455206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.451817 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010ed3-openshift-default-dockercfg-9tvsf\"" Nov 03 19:03:27.500368 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.500339 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:03:27.527994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.527964 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528214 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528018 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528214 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528055 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528214 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528086 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528214 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528159 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528214 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528181 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528234 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528257 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvskz\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528331 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528705 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528677 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.528988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.528947 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.529089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.529067 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.529414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.529390 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.529582 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.529559 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.531175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.531149 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.531588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.531571 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.587572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.587534 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvskz\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.616681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.616645 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:27.647470 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.647427 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:03:27.716009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:27.715927 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:28.064311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.064231 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" event={"ID":"b3906bfb-afdb-4752-907f-695607248522","Type":"ContainerStarted","Data":"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb"} Nov 03 19:03:28.064311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.064297 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" event={"ID":"b3906bfb-afdb-4752-907f-695607248522","Type":"ContainerStarted","Data":"3479956040f08eb798b1df436258c1e4af63919ee43c89de1bac1b66fc37b15c"} Nov 03 19:03:28.158404 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:28.158351 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dd13a0_d8e0_4b18_886a_e914300015ac.slice/crio-2923b120b5cd52a0b58325df1ab115973a6b9eda87aa462709c2b42a14478e67 WatchSource:0}: Error finding container 2923b120b5cd52a0b58325df1ab115973a6b9eda87aa462709c2b42a14478e67: Status 404 returned error can't find the container with id 2923b120b5cd52a0b58325df1ab115973a6b9eda87aa462709c2b42a14478e67 Nov 03 19:03:28.163332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.162646 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:28.163332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.162731 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:28.163332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.162769 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:28.168821 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.167873 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:03:28.211033 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.210958 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podStartSLOduration=2.210935836 podStartE2EDuration="2.210935836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:28.181299903 +0000 UTC m=+1089.592400088" watchObservedRunningTime="2026-11-03 19:03:28.210935836 +0000 UTC m=+1089.622035965" Nov 03 19:03:28.731542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.731511 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:28.732822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.732799 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:28.732877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:28.732849 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:29.071949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.071824 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" event={"ID":"39dd13a0-d8e0-4b18-886a-e914300015ac","Type":"ContainerStarted","Data":"2892bbf248f4101f07febe3709873c546010f05c7c55e883f392f613ee47f42d"} Nov 03 19:03:29.071949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.071878 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" event={"ID":"39dd13a0-d8e0-4b18-886a-e914300015ac","Type":"ContainerStarted","Data":"2923b120b5cd52a0b58325df1ab115973a6b9eda87aa462709c2b42a14478e67"} Nov 03 19:03:29.242167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.242100 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podStartSLOduration=2.242085178 podStartE2EDuration="2.242085178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:29.239630732 +0000 UTC m=+1090.650730855" watchObservedRunningTime="2026-11-03 19:03:29.242085178 +0000 UTC m=+1090.653185301" Nov 03 19:03:29.323207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.321629 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:03:29.331676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.331610 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350271 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350336 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350376 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350455 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350556 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350531 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350556 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350549 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350628 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350575 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqdz\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350663 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350628 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.350703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.350681 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.439906 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.439867 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:03:29.451871 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.451841 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.451888 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.451924 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.451940 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.451969 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452016 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452033 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452388 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452058 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqdz\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452388 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452091 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452462 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.452672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452649 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.453170 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452829 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.453170 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.452873 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.454047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.454023 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.455698 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.455669 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.456410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.456387 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.530667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.530632 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqdz\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.531425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.531401 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-99d5p\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.646607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.646573 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:29.729082 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.728999 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:29.729703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.729385 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:29.729703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.729435 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:29.735580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.735465 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:29.735580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:29.735520 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:30.135199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.135166 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:03:30.144250 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:03:30.144217 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce96116_b75b_4735_bdd9_e31a03f583fb.slice/crio-6e4c2e03ba6403f6eba3723ee626391eca3b547a64acb1f11165b1d340361d8a WatchSource:0}: Error finding container 6e4c2e03ba6403f6eba3723ee626391eca3b547a64acb1f11165b1d340361d8a: Status 404 returned error can't find the container with id 6e4c2e03ba6403f6eba3723ee626391eca3b547a64acb1f11165b1d340361d8a Nov 03 19:03:30.146925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.146894 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:30.147000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.146958 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:30.147000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.146988 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:30.724144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.722242 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:30.724144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.722315 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:30.732553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.732519 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:30.732751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:30.732574 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:31.147143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.146160 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" event={"ID":"5ce96116-b75b-4735-bdd9-e31a03f583fb","Type":"ContainerStarted","Data":"b86838ebfd999d2fd0af9831b43256c7561109c47e54224981c4bdcbe2ba21ca"} Nov 03 19:03:31.147143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.146229 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" event={"ID":"5ce96116-b75b-4735-bdd9-e31a03f583fb","Type":"ContainerStarted","Data":"6e4c2e03ba6403f6eba3723ee626391eca3b547a64acb1f11165b1d340361d8a"} Nov 03 19:03:31.262741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.262687 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podStartSLOduration=2.262673639 podStartE2EDuration="2.262673639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:31.260521628 +0000 UTC m=+1092.671621751" watchObservedRunningTime="2026-11-03 19:03:31.262673639 +0000 UTC m=+1092.673773762" Nov 03 19:03:31.650398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.650295 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:31.651255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.651226 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.651483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.651455 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:31.721108 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.721068 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.721561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.721514 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:31.738610 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.738496 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.738610 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:31.738553 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:32.023503 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:03:32.023396 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:03:32.647696 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.647661 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.648368 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.648323 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:32.726154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.725008 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.726154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.725069 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:32.739563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.739528 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.739793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:32.739768 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:33.647674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.647626 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.647849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.647688 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:33.719597 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.719563 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.719845 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.719820 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:33.733548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.733503 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.733864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:33.733792 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 19:03:34.647750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.647715 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:34.647928 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.647769 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:34.716568 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.716535 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 19:03:34.716754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.716584 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 19:03:34.735509 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.735482 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:34.735853 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.735831 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:34.736578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:34.736558 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:03:35.647767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:35.647735 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:35.647947 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:35.647786 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:35.720474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:35.720444 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:35.720743 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:35.720724 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:35.721471 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:35.721450 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:03:36.646991 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:36.646950 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:36.647190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:36.647019 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:37.647539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:37.647505 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-99d5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:03:37.647970 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:37.647588 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 19:03:38.902729 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:38.902696 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:38.903610 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:38.903591 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:38.904310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:38.904294 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:03:59.818882 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:03:59.818845 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:04:01.937476 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:04:01.937434 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:04:28.533960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.533925 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:04:28.539286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.539264 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.588354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.588330 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-92596a-openshift-default-dockercfg-zf4cq\"" Nov 03 19:04:28.638503 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.638476 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:04:28.658884 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.658850 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659033 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.658899 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659045 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpn5k\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659079 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659099 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659232 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659220 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659304 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659403 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.659465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.659437 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761008 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761045 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761086 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761106 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761196 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpn5k\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761235 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761265 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761334 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761367 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761499 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761663 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.761955 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.761933 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.762041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.762002 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.762103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.762077 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.763550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.763530 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.763730 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.763713 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.819808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.819744 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.819925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.819882 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpn5k\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:28.852801 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:28.852769 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:29.279414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.279365 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:04:29.283864 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:04:29.283832 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f6ba93_1fb3_4743_ae83_0c856a53d2b8.slice/crio-407b35ad832ab5ec4ac997479e8afdbd591af515fa695c3d09ad556d7111c43c WatchSource:0}: Error finding container 407b35ad832ab5ec4ac997479e8afdbd591af515fa695c3d09ad556d7111c43c: Status 404 returned error can't find the container with id 407b35ad832ab5ec4ac997479e8afdbd591af515fa695c3d09ad556d7111c43c Nov 03 19:04:29.286327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.286292 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:29.286407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.286369 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:29.286449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.286411 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:29.602866 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.602826 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" event={"ID":"25f6ba93-1fb3-4743-ae83-0c856a53d2b8","Type":"ContainerStarted","Data":"bf7005bf7a863799407dc8f043156c232f66bdbda0b1ee1958a8a09dab054ea1"} Nov 03 19:04:29.603267 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.602895 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" event={"ID":"25f6ba93-1fb3-4743-ae83-0c856a53d2b8","Type":"ContainerStarted","Data":"407b35ad832ab5ec4ac997479e8afdbd591af515fa695c3d09ad556d7111c43c"} Nov 03 19:04:29.718643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.718581 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podStartSLOduration=1.718563321 podStartE2EDuration="1.718563321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:29.715795931 +0000 UTC m=+1151.126896057" watchObservedRunningTime="2026-11-03 19:04:29.718563321 +0000 UTC m=+1151.129663445" Nov 03 19:04:29.853707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:29.853618 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:30.858193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:30.858148 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:30.858696 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:30.858226 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:31.853695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:31.853658 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:31.853885 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:31.853711 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:31.940514 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:04:31.940470 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:04:32.306626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:32.306535 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:04:32.853504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:32.853465 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:32.853712 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:32.853529 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:33.859141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:33.857551 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:33.859141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:33.857626 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:34.853559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:34.853519 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:34.853715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:34.853580 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:35.854046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:35.854011 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:04:35.854474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:35.854083 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 19:04:36.857393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:36.857360 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:36.857782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:36.857636 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:36.858441 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:36.858422 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:04:41.665773 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.665740 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:04:41.669989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.669970 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.769323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.769269 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:04:41.769895 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.769860 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.769932 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.769974 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770002 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770035 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2xr\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770165 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770212 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770284 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.770427 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.770323 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871586 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871549 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871611 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871670 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871705 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871753 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871811 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871855 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871886 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.871997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.871917 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2xr\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.872342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.872312 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.872474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.872345 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.872474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.872451 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.872474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.872466 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.872639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.872583 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.874321 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.874301 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.874699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.874677 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.928912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.928839 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.929032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.928939 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2xr\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:41.983541 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:41.983516 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:42.301184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.301083 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:04:42.305980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.305961 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376347 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376394 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376424 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376596 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376633 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376694 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376748 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376771 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.376967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.376874 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfk7\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.406358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.406325 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:04:42.410922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.410893 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:04:42.415790 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:04:42.415764 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4797a113_bf51_4ed9_a506_a00be7473743.slice/crio-bb688e833f4b88079bcc67ef80f46d40b45dd3d9669b47088b18e274227eb1c7 WatchSource:0}: Error finding container bb688e833f4b88079bcc67ef80f46d40b45dd3d9669b47088b18e274227eb1c7: Status 404 returned error can't find the container with id bb688e833f4b88079bcc67ef80f46d40b45dd3d9669b47088b18e274227eb1c7 Nov 03 19:04:42.418802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.418766 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:42.418872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.418851 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:42.418914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.418895 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:42.478269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478244 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478274 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478294 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478339 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478363 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478403 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfk7\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478505 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478551 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478599 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478763 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478649 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478763 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478738 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.478951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.478925 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.479065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.479043 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.479065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.479059 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.480916 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.480886 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.481003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.480899 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.536580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.536550 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.537337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.537313 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfk7\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.619932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.619901 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:42.705265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.705224 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" event={"ID":"4797a113-bf51-4ed9-a506-a00be7473743","Type":"ContainerStarted","Data":"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1"} Nov 03 19:04:42.705689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.705292 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" event={"ID":"4797a113-bf51-4ed9-a506-a00be7473743","Type":"ContainerStarted","Data":"bb688e833f4b88079bcc67ef80f46d40b45dd3d9669b47088b18e274227eb1c7"} Nov 03 19:04:42.820709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.820660 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podStartSLOduration=1.8206458049999998 podStartE2EDuration="1.820645805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:42.817774692 +0000 UTC m=+1164.228874816" watchObservedRunningTime="2026-11-03 19:04:42.820645805 +0000 UTC m=+1164.231745988" Nov 03 19:04:42.983774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:42.983691 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:43.048953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.048825 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:04:43.055012 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:04:43.054981 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96737262_e6e0_4294_a311_04f0f0475bae.slice/crio-0e87aad575885aeb716962b32a638fc11aeac10e04a644fdc57df91a70b1cc16 WatchSource:0}: Error finding container 0e87aad575885aeb716962b32a638fc11aeac10e04a644fdc57df91a70b1cc16: Status 404 returned error can't find the container with id 0e87aad575885aeb716962b32a638fc11aeac10e04a644fdc57df91a70b1cc16 Nov 03 19:04:43.057677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.057644 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:43.057774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.057710 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:43.057774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.057738 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:43.355050 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.355010 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:04:43.358857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.358835 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.419160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.417960 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512091 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512175 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512225 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512248 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512286 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512325 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512427 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512508 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.513207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.512532 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrsh\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614173 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614227 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614268 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614283 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614320 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614467 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614752 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614723 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.614922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.614735 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615213 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615329 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615424 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615456 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrsh\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615651 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615555 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.615781 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.615757 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.616864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.616843 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.617201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.617180 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.671608 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.671584 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.672296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.672272 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrsh\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.673083 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.673062 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:43.719507 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.719470 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" event={"ID":"96737262-e6e0-4294-a311-04f0f0475bae","Type":"ContainerStarted","Data":"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe"} Nov 03 19:04:43.719831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.719537 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" event={"ID":"96737262-e6e0-4294-a311-04f0f0475bae","Type":"ContainerStarted","Data":"0e87aad575885aeb716962b32a638fc11aeac10e04a644fdc57df91a70b1cc16"} Nov 03 19:04:43.837000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.835930 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podStartSLOduration=1.835915425 podStartE2EDuration="1.835915425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:43.833284089 +0000 UTC m=+1165.244384214" watchObservedRunningTime="2026-11-03 19:04:43.835915425 +0000 UTC m=+1165.247015548" Nov 03 19:04:43.994984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.994891 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:43.995270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:43.995237 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:44.139966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.139928 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:04:44.147031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.146998 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:44.147134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.147066 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:44.147134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.147093 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:44.633142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.630496 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:44.636176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.634221 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:44.636176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.634288 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:44.773436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.773395 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" event={"ID":"361ba76d-8a0b-4e71-8240-525f86766a44","Type":"ContainerStarted","Data":"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150"} Nov 03 19:04:44.773807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.773469 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" event={"ID":"361ba76d-8a0b-4e71-8240-525f86766a44","Type":"ContainerStarted","Data":"4d37034507e143b9de03b22816c5b0038fba632de8aa333e7e13c2e84830f362"} Nov 03 19:04:44.999237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.997475 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:44.999237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:44.997531 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:45.647144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.637686 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:45.647144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.637755 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:45.674268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.674237 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:45.675746 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.675723 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:45.675922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.675897 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:45.986199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.984516 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:45.986199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:45.984576 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:46.640146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.632042 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:46.640146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.632103 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:46.673432 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.673393 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:46.673618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.673438 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:46.987590 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.987502 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:47.007842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:46.997243 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:47.934143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:47.933409 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:47.934143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:47.933500 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:48.021302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:47.996110 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.022009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.021954 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:48.159171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.141563 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.159171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.142085 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:48.211751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.211646 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.675378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.675325 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.786652 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.786994 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.800357 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.801717 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.801773 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.817748 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.817800 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:48.876148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.837935 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:04:48.914409 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.898258 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.914409 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.899879 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.936315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.936218 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:04:48.969153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.967276 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:48.987129 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.987083 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.987402 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:48.987373 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 19:04:49.620589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:49.620543 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:04:49.621220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:49.621187 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 19:04:49.673598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:49.673559 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:49.673881 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:49.673852 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:50.294144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.293438 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:50.301141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.294969 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:50.303369 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.303337 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:04:50.507141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.506235 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podStartSLOduration=7.506216513 podStartE2EDuration="7.506216513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:44.91248833 +0000 UTC m=+1166.323588477" watchObservedRunningTime="2026-11-03 19:04:50.506216513 +0000 UTC m=+1171.917316637" Nov 03 19:04:50.673999 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.673959 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:50.674500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.674015 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:50.896562 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.896530 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:50.898269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.898249 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:50.906479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:50.904302 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:04:51.674133 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:51.674087 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:04:51.674507 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:51.674160 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 19:04:52.690987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:52.690950 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:52.699517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:52.699495 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:04:52.699676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:04:52.699554 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:05:01.957545 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:05:01.957459 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:05:03.018238 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:03.018199 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:05:12.831893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.831859 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:05:12.835730 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.835713 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.929992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.929935 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930001 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930321 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930241 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930321 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930289 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930338 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930417 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lssl\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930442 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930488 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:12.930548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:12.930520 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.029838 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.029800 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:05:13.031794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.031752 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lssl\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.031967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.031797 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.031967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.031859 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.031967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.031900 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.031967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.031966 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.032346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.032016 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.032346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.032110 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.032346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.032168 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.032346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.032237 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.032824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.032799 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.033419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.033397 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.033611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.033574 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.033816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.033798 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.033919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.033871 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.035813 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.035759 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.036608 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.036585 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.089397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.089325 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lssl\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.090811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.090791 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.148674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.148620 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:13.573755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.573720 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:05:13.579943 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:05:13.579917 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e09a625_6a77_44a6_952a_41846b97e38d.slice/crio-77aa63e73f69c3dfda8b5e35b780807fda40b4c9574f4b4ba1ad5796da002ce5 WatchSource:0}: Error finding container 77aa63e73f69c3dfda8b5e35b780807fda40b4c9574f4b4ba1ad5796da002ce5: Status 404 returned error can't find the container with id 77aa63e73f69c3dfda8b5e35b780807fda40b4c9574f4b4ba1ad5796da002ce5 Nov 03 19:05:13.582359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.582326 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:13.582438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.582392 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:13.582438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.582419 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:13.982963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.982870 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" event={"ID":"7e09a625-6a77-44a6-952a-41846b97e38d","Type":"ContainerStarted","Data":"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f"} Nov 03 19:05:13.982963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:13.982929 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" event={"ID":"7e09a625-6a77-44a6-952a-41846b97e38d","Type":"ContainerStarted","Data":"77aa63e73f69c3dfda8b5e35b780807fda40b4c9574f4b4ba1ad5796da002ce5"} Nov 03 19:05:14.148874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:14.148843 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:14.150553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:14.150527 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:14.150684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:14.150570 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:14.151949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:14.151903 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podStartSLOduration=2.151891097 podStartE2EDuration="2.151891097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:14.148524204 +0000 UTC m=+1195.559624344" watchObservedRunningTime="2026-11-03 19:05:14.151891097 +0000 UTC m=+1195.562991247" Nov 03 19:05:15.149554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:15.149520 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:15.149965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:15.149573 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:16.149252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:16.149217 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:16.149449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:16.149269 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:17.149243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:17.149194 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:17.149655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:17.149267 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:18.150031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:18.149994 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:18.150561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:18.150050 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:19.148978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:19.148937 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:19.149200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:19.149007 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:20.149147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:20.149079 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:05:20.149609 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:20.149156 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 19:05:21.153078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:21.153046 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:21.155449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:21.153361 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:21.155449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:21.154070 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:05:27.318368 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.318332 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:05:27.322380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.322359 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.361995 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.361955 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362004 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362088 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362192 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362376 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362236 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh279\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362376 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362304 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362376 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362360 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362513 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362409 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.362584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.362564 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464331 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464428 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464464 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464495 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464522 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464570 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464615 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh279\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464641 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.464723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.464675 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.465152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.465113 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.465622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.465599 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.466421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.466221 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.466421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.466364 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.467371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.467341 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.467949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.467929 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.468915 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.468706 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.514438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.514403 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:05:27.522993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.522962 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.524246 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.524223 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh279\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:27.635910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:27.635872 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:28.067554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:28.067468 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:05:28.072747 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:05:28.072720 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42d275d_beb4_4789_861b_83632fbba289.slice/crio-33828bccdb18520d420579865bc63df00bc8fa2274599f984ef159bbbaae0e0f WatchSource:0}: Error finding container 33828bccdb18520d420579865bc63df00bc8fa2274599f984ef159bbbaae0e0f: Status 404 returned error can't find the container with id 33828bccdb18520d420579865bc63df00bc8fa2274599f984ef159bbbaae0e0f Nov 03 19:05:28.075289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:28.075260 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:28.075374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:28.075325 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:28.075374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:28.075352 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:28.092439 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:28.092411 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" event={"ID":"b42d275d-beb4-4789-861b-83632fbba289","Type":"ContainerStarted","Data":"33828bccdb18520d420579865bc63df00bc8fa2274599f984ef159bbbaae0e0f"} Nov 03 19:05:29.105454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:29.105412 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" event={"ID":"b42d275d-beb4-4789-861b-83632fbba289","Type":"ContainerStarted","Data":"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7"} Nov 03 19:05:29.270875 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:29.270822 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podStartSLOduration=2.270808413 podStartE2EDuration="2.270808413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:29.268416317 +0000 UTC m=+1210.679516460" watchObservedRunningTime="2026-11-03 19:05:29.270808413 +0000 UTC m=+1210.681908535" Nov 03 19:05:29.636508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:29.636470 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:29.638161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:29.638139 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:29.638260 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:29.638183 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:30.636345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:30.636307 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:30.636740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:30.636380 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:31.637005 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:31.636968 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:31.637434 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:31.637041 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:31.938515 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:05:31.938421 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:05:32.636371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:32.636333 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:32.636614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:32.636402 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:33.636776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:33.636738 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:33.637223 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:33.636804 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:34.637202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:34.637161 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:05:34.637589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:34.637228 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 19:05:35.640894 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:35.640856 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:35.641357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:35.641227 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:35.642103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:35.642082 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:05:40.379655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.379619 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:05:40.799411 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.799317 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:05:40.803246 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.803229 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.883380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.883346 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:05:40.901649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.901620 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"distinct-2-w0-92596a-kserve-self-signed-certs\"" Nov 03 19:05:40.956869 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.956835 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.957031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.956876 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdqd\" (UniqueName: \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.957031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.956962 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.957110 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.957083 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.957230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.957217 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:40.957277 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:40.957263 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058074 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058133 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdqd\" (UniqueName: \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058167 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058245 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058363 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058393 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058730 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058706 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058748 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.058815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.058791 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.060449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.060419 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.060777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.060756 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.115982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.115949 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdqd\" (UniqueName: \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\") pod \"distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.116216 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.116202 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:41.500233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:41.500200 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:05:41.503249 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:05:41.503222 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1922c791_3b1f_4594_84cc_1bb90fb16d52.slice/crio-65e1d274f2094597fad48d19218d183fc4d76bf241abb89af9395bced3580fff WatchSource:0}: Error finding container 65e1d274f2094597fad48d19218d183fc4d76bf241abb89af9395bced3580fff: Status 404 returned error can't find the container with id 65e1d274f2094597fad48d19218d183fc4d76bf241abb89af9395bced3580fff Nov 03 19:05:42.200142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:42.199783 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" event={"ID":"1922c791-3b1f-4594-84cc-1bb90fb16d52","Type":"ContainerStarted","Data":"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8"} Nov 03 19:05:42.200142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:42.199835 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" event={"ID":"1922c791-3b1f-4594-84cc-1bb90fb16d52","Type":"ContainerStarted","Data":"65e1d274f2094597fad48d19218d183fc4d76bf241abb89af9395bced3580fff"} Nov 03 19:05:42.322032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:42.321963 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podStartSLOduration=2.321943036 podStartE2EDuration="2.321943036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:42.318596563 +0000 UTC m=+1223.729696711" watchObservedRunningTime="2026-11-03 19:05:42.321943036 +0000 UTC m=+1223.733043161" Nov 03 19:05:43.116748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:43.116720 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:51.116440 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:51.116407 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:51.129438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:51.129409 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:05:51.269352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:05:51.269322 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:06:01.939293 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:06:01.939253 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:06:23.853813 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:23.853775 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:06:31.938230 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:06:31.938099 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:06:40.619268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.619233 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:06:40.623399 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.623379 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.679667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.679634 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:06:40.721485 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.721455 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"distinct-2-w2-619731-kserve-self-signed-certs\"" Nov 03 19:06:40.729155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729113 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.729300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729243 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.729300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729271 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.729373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729304 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.729373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729325 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282jc\" (UniqueName: \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.729479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.729459 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.830788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.830751 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.830996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.830978 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831086 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831113 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831163 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831188 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282jc\" (UniqueName: \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831466 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831438 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831601 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.831755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.831687 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.833636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.833592 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.833796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.833776 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.889406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.889335 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282jc\" (UniqueName: \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\") pod \"distinct-2-w2-619731-kserve-56b9889d5c-jgwmw\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:40.936065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:40.936030 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:41.315716 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:41.315621 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:06:41.321172 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:06:41.321144 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcc1b9ea_a25e_431f_a3d4_7e5f28577346.slice/crio-3707ea38f691e15892af328ee58e8e7ad55cce6257113f9324a76e373d6a6cac WatchSource:0}: Error finding container 3707ea38f691e15892af328ee58e8e7ad55cce6257113f9324a76e373d6a6cac: Status 404 returned error can't find the container with id 3707ea38f691e15892af328ee58e8e7ad55cce6257113f9324a76e373d6a6cac Nov 03 19:06:41.651200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:41.651161 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" event={"ID":"bcc1b9ea-a25e-431f-a3d4-7e5f28577346","Type":"ContainerStarted","Data":"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2"} Nov 03 19:06:41.651580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:41.651212 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" event={"ID":"bcc1b9ea-a25e-431f-a3d4-7e5f28577346","Type":"ContainerStarted","Data":"3707ea38f691e15892af328ee58e8e7ad55cce6257113f9324a76e373d6a6cac"} Nov 03 19:06:41.763298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:41.763239 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podStartSLOduration=1.763226005 podStartE2EDuration="1.763226005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:41.7604517 +0000 UTC m=+1283.171551824" watchObservedRunningTime="2026-11-03 19:06:41.763226005 +0000 UTC m=+1283.174326128" Nov 03 19:06:41.936834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:41.936742 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:43.853664 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.853630 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:06:43.857659 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.857641 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.900128 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.900084 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:06:43.903966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.903942 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.952831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.952796 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:06:43.985006 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.984969 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985016 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985193 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985240 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985266 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985308 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985335 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985357 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srzp\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985410 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985435 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxqw\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985457 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985489 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985515 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985538 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985571 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:43.985967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985749 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985772 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:43.985967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:43.985859 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.015787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.015753 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:06:44.086889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.086857 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.086889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.086891 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxqw\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.086911 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087022 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087056 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087079 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087112 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087282 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087316 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087361 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087401 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087432 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087451 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087506 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087527 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087567 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087571 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087592 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087645 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087673 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087698 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srzp\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.087965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087831 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.087965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.087912 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.088304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.088013 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.088304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.088211 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.088418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.088304 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.088490 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.088468 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.088781 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.088756 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.090484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.090461 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.091067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.091042 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.091067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.091056 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.091257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.091055 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.144269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.144203 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxqw\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.148460 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.148434 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.148783 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.148760 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.148835 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.148818 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srzp\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.171861 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.171837 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:44.217066 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.217024 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:44.606846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.606816 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:06:44.611541 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:06:44.611511 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a521666_3127_4542_95b0_0d51a945815d.slice/crio-276b88a01aabfa2cc9fcbd2f087b8defc168bd882f169c1df7b1f80af963dc5e WatchSource:0}: Error finding container 276b88a01aabfa2cc9fcbd2f087b8defc168bd882f169c1df7b1f80af963dc5e: Status 404 returned error can't find the container with id 276b88a01aabfa2cc9fcbd2f087b8defc168bd882f169c1df7b1f80af963dc5e Nov 03 19:06:44.613831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.613796 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.613930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.613862 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.613930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.613892 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.652369 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.652346 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:06:44.661215 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:06:44.661187 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93f4c25_cc2a_4123_90c7_98eec6b3a0cb.slice/crio-20feffd7b2c6f31292a136198b316af43931fcfbd0df57214d703fc52c28fd6e WatchSource:0}: Error finding container 20feffd7b2c6f31292a136198b316af43931fcfbd0df57214d703fc52c28fd6e: Status 404 returned error can't find the container with id 20feffd7b2c6f31292a136198b316af43931fcfbd0df57214d703fc52c28fd6e Nov 03 19:06:44.663984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.663952 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.664070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.664026 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.664070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.664055 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:44.675347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.675320 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" event={"ID":"3a521666-3127-4542-95b0-0d51a945815d","Type":"ContainerStarted","Data":"276b88a01aabfa2cc9fcbd2f087b8defc168bd882f169c1df7b1f80af963dc5e"} Nov 03 19:06:44.676683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:44.676648 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" event={"ID":"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb","Type":"ContainerStarted","Data":"20feffd7b2c6f31292a136198b316af43931fcfbd0df57214d703fc52c28fd6e"} Nov 03 19:06:45.733720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:45.733677 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" event={"ID":"3a521666-3127-4542-95b0-0d51a945815d","Type":"ContainerStarted","Data":"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a"} Nov 03 19:06:45.735850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:45.735818 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" event={"ID":"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb","Type":"ContainerStarted","Data":"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392"} Nov 03 19:06:45.960569 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:45.960516 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podStartSLOduration=2.960500163 podStartE2EDuration="2.960500163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:45.913598588 +0000 UTC m=+1287.324698714" watchObservedRunningTime="2026-11-03 19:06:45.960500163 +0000 UTC m=+1287.371600364" Nov 03 19:06:46.048261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.048164 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podStartSLOduration=3.048149818 podStartE2EDuration="3.048149818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:46.043635291 +0000 UTC m=+1287.454735431" watchObservedRunningTime="2026-11-03 19:06:46.048149818 +0000 UTC m=+1287.459249946" Nov 03 19:06:46.172859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.172823 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:46.174279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.174255 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:46.174389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.174302 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:46.221363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.219491 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:46.227810 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.227774 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:46.228068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:46.228038 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:47.172201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:47.172163 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:47.172692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:47.172226 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:47.225460 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:47.225276 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:47.225460 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:47.225393 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:48.172643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:48.172610 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:48.173061 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:48.172663 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:48.221157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:48.220477 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:48.221157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:48.220540 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:49.172216 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:49.172179 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:49.172395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:49.172241 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:49.224144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:49.221485 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:49.224144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:49.221543 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:50.172482 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.172446 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:50.172661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.172518 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:50.218144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.217519 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:50.218144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.217600 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:50.936533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.936495 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:50.950987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:50.950956 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:51.172531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:51.172491 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:06:51.172725 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:51.172548 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 19:06:51.217980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:51.217895 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:06:51.217980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:51.217955 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 19:06:51.856680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:51.856643 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:06:52.202037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.201949 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:52.202549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.202526 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:52.203536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.203516 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:06:52.221917 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.221890 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:52.222217 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.222193 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:06:52.222791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:06:52.222773 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:07:01.940200 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:07:01.940163 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:07:02.145474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:02.145440 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:07:04.458318 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:04.458261 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:04.534259 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:04.534212 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:04.539237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:04.539193 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:04.845592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:04.845545 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:28.844134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:28.844097 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:07:28.849324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:28.849302 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:28.933371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:28.933336 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:07:29.017765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.017732 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.017927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.017769 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.017927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.017877 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.017965 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018034 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.017995 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.018049 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8jd\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.018112 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.018188 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.018233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.018209 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.118950 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.118869 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.118950 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.118935 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.118965 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119041 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119068 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119200 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119266 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119306 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119391 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8jd\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119457 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119492 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119562 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.119914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.119888 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.120085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.120066 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.121330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.121303 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.121628 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.121607 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.176800 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.176770 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.176917 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.176878 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8jd\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.463132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.463038 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:29.894193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.894159 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:07:29.902864 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:07:29.902836 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376ad674_5c3d_49e7_9697_2dc47b219634.slice/crio-bb4484b11cbc42bcf29fcea08dcf6dd92079f5687c711c7b74d161784f53b624 WatchSource:0}: Error finding container bb4484b11cbc42bcf29fcea08dcf6dd92079f5687c711c7b74d161784f53b624: Status 404 returned error can't find the container with id bb4484b11cbc42bcf29fcea08dcf6dd92079f5687c711c7b74d161784f53b624 Nov 03 19:07:29.905451 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.905417 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:29.905539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.905488 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:29.905539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.905526 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:29.940020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.939988 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:07:29.945812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.945793 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:29.990490 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:29.990456 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:07:30.033524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033247 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033435 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033509 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgzn\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033542 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033568 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033604 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033686 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.033811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033744 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.034150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.033850 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.110079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.110041 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" event={"ID":"376ad674-5c3d-49e7-9697-2dc47b219634","Type":"ContainerStarted","Data":"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481"} Nov 03 19:07:30.110266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.110104 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" event={"ID":"376ad674-5c3d-49e7-9697-2dc47b219634","Type":"ContainerStarted","Data":"bb4484b11cbc42bcf29fcea08dcf6dd92079f5687c711c7b74d161784f53b624"} Nov 03 19:07:30.135358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135327 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135521 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135383 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135521 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135470 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135521 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgzn\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135665 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135812 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135858 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.135902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135893 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136015 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.135984 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136072 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.136039 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136072 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.136051 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.136228 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.136314 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.136445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.136429 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.137740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.137719 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.138534 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.138514 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.195540 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.195469 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgzn\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.195725 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.195707 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.247717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.247648 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podStartSLOduration=2.247633488 podStartE2EDuration="2.247633488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:30.246162808 +0000 UTC m=+1331.657262939" watchObservedRunningTime="2026-11-03 19:07:30.247633488 +0000 UTC m=+1331.658733611" Nov 03 19:07:30.271044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.271012 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:30.463976 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.463887 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:30.465428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.465377 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:30.465428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.465423 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:30.744771 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.744679 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:07:30.750021 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:07:30.749990 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb44d65f_6697_4923_9fb5_7d5f8a89afb8.slice/crio-265bb57a20c7709ea24fcdf2737bf57c7fc5ff6f7603f404a58e89b14049480c WatchSource:0}: Error finding container 265bb57a20c7709ea24fcdf2737bf57c7fc5ff6f7603f404a58e89b14049480c: Status 404 returned error can't find the container with id 265bb57a20c7709ea24fcdf2737bf57c7fc5ff6f7603f404a58e89b14049480c Nov 03 19:07:30.752560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.752527 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:30.752645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.752589 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:30.752645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:30.752618 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:31.119855 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.119798 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" event={"ID":"eb44d65f-6697-4923-9fb5-7d5f8a89afb8","Type":"ContainerStarted","Data":"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480"} Nov 03 19:07:31.120360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.119889 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" event={"ID":"eb44d65f-6697-4923-9fb5-7d5f8a89afb8","Type":"ContainerStarted","Data":"265bb57a20c7709ea24fcdf2737bf57c7fc5ff6f7603f404a58e89b14049480c"} Nov 03 19:07:31.235248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.235184 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podStartSLOduration=2.235163559 podStartE2EDuration="2.235163559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:31.233690258 +0000 UTC m=+1332.644790439" watchObservedRunningTime="2026-11-03 19:07:31.235163559 +0000 UTC m=+1332.646263686" Nov 03 19:07:31.271408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.271381 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:31.272928 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.272902 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:31.273062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.272948 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:31.464380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.464295 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:31.464380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:31.464358 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:31.953655 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:07:31.953616 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:07:32.271475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:32.271396 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:32.271475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:32.271445 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:32.464288 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:32.464259 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:32.464524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:32.464504 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:33.272291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:33.272258 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:33.272759 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:33.272312 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:33.464334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:33.464299 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:33.464528 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:33.464359 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:34.272182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:34.272153 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:34.272390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:34.272369 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:34.464136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:34.463749 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:34.464136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:34.463796 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:35.275544 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:35.275509 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:35.276147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:35.276091 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:35.465502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:35.465467 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:35.465780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:35.465751 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:36.282145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.272501 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:36.282145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.272567 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:36.376152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.372158 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:07:36.481094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.481055 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:36.537037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.531099 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:36.537037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.536785 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:07:36.537037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.536936 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.542337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.542305 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.594746 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.594318 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"distinct-2-w3-fa2878-kserve-self-signed-certs\"" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.657652 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.658313 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.658474 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.658600 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.658805 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xk5\" (UniqueName: \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.659140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.658938 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.831947 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.832216 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.832258 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.832282 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.832349 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xk5\" (UniqueName: \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.832384 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.833059 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.852141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.833348 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.871675 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.871632 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.876157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.876055 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.876397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.872525 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.902499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.902420 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 03 19:07:36.966570 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.941739 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:36.997141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:36.988490 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:37.003009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.002974 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xk5\" (UniqueName: \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\") pod \"distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:37.231193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.230536 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:37.403210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.392241 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:37.403210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.392315 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:37.547587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.547535 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.548886 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.548714 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:07:37.570539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.570506 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:37.570802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.570769 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:37.635277 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.635229 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.789355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.789246 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.884425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.884380 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.896671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.896626 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.928141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.898640 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:07:37.931105 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.931068 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:07:37.997147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:37.980858 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.121258 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.121216 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.289145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.273142 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:38.289145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.273204 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:38.394963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.394868 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.395300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.395207 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.464271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.463255 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.527252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.527218 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:38.527559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.527506 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:38.565406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.557816 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.640799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.640753 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.642309 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.641110 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:07:38.681653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.681511 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 19:07:38.785157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.767198 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 03 19:07:38.808315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.807513 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.813799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.813756 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:07:38.886297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.886248 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.915237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.915192 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:38.994282 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.994182 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:07:38.994838 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:38.994817 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:07:39.103676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.103647 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:07:39.213289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.122347 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.213685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.213656 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:07:39.324555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.324526 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:39.324811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.324783 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:39.468602 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.468547 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:07:39.470906 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.470820 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 19:07:39.472099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.471930 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.478722 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.478700 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:07:39.728142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.711453 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 19:07:39.748567 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.748536 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:07:39.783140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.749620 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.783642 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.773900 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.858189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.781242 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:07:39.858189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.838771 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:07:39.898161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.870974 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:07:39.898161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.894209 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.901014 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:39.900988 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:07:40.061709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.061623 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:07:40.062541 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.062509 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:07:40.067140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.063551 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:07:40.079931 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.079906 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:07:40.276269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.276240 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:07:40.276570 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.276519 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 19:07:40.421805 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.421776 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:07:40.425350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.425323 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:07:40.751456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.751355 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" event={"ID":"30838aaa-f1bc-4076-b3da-0b79dbe9d30b","Type":"ContainerStarted","Data":"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6"} Nov 03 19:07:40.751456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.751405 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" event={"ID":"30838aaa-f1bc-4076-b3da-0b79dbe9d30b","Type":"ContainerStarted","Data":"3c98d241c1b0d19a31f6d86a2fe32dd2ad295eb909b83819f404854c74fd574e"} Nov 03 19:07:40.884369 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.884315 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podStartSLOduration=4.884301455 podStartE2EDuration="4.884301455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:40.884096971 +0000 UTC m=+1342.295197095" watchObservedRunningTime="2026-11-03 19:07:40.884301455 +0000 UTC m=+1342.295401577" Nov 03 19:07:40.951723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.951675 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:40.952225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.952189 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:40.953569 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:40.953544 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:07:41.232167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:41.232135 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:42.313305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:42.313262 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:42.313770 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:42.313321 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:42.709654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:42.709623 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:42.710590 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:42.710507 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:42.711287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:42.711262 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:07:45.008155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.008109 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:07:45.012386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.012365 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.053712 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.053684 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s297f\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.053849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.053735 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.053849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.053813 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.053951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.053863 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.053951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.053883 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.054026 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.054012 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.054063 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.054045 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.054107 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.054064 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.054196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.054187 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.078578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.078551 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:07:45.156001 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.155968 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s297f\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156024 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156059 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156098 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156165 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156412 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156262 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156412 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156293 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156525 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156411 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156525 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156482 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156514 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156613 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156691 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156771 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.156865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.156842 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.158630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.158610 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.158733 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.158610 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.213737 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.213710 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.213944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.213925 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s297f\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.327576 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.327542 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:45.756852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.756763 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:07:45.760578 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:07:45.760553 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dc0ad5_7047_4668_8563_eb5242bb96ec.slice/crio-c90162b15fe70a1529b0280e311b3f337c855c39e371e692286dfd4f70e7063e WatchSource:0}: Error finding container c90162b15fe70a1529b0280e311b3f337c855c39e371e692286dfd4f70e7063e: Status 404 returned error can't find the container with id c90162b15fe70a1529b0280e311b3f337c855c39e371e692286dfd4f70e7063e Nov 03 19:07:45.763163 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.763112 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:45.763254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.763199 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:45.763254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.763227 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:45.803280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:45.803215 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" event={"ID":"22dc0ad5-7047-4668-8563-eb5242bb96ec","Type":"ContainerStarted","Data":"c90162b15fe70a1529b0280e311b3f337c855c39e371e692286dfd4f70e7063e"} Nov 03 19:07:46.817503 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:46.817463 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" event={"ID":"22dc0ad5-7047-4668-8563-eb5242bb96ec","Type":"ContainerStarted","Data":"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618"} Nov 03 19:07:46.997142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:46.990684 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podStartSLOduration=2.990664568 podStartE2EDuration="2.990664568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:46.988393813 +0000 UTC m=+1348.399493936" watchObservedRunningTime="2026-11-03 19:07:46.990664568 +0000 UTC m=+1348.401764692" Nov 03 19:07:47.231720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.231682 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:47.245541 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.245510 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:47.328749 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.328714 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:47.330560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.330534 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:47.330698 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.330597 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:47.835699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:47.835670 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:07:48.328189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:48.328157 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:48.328374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:48.328213 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:49.328303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:49.328261 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:49.328774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:49.328323 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:50.328833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:50.328783 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:50.329253 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:50.328860 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:51.328103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:51.328067 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:51.328290 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:51.328159 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:52.328908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:52.328876 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:07:52.329310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:52.328923 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 19:07:53.651394 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:53.651360 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:53.652839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:53.652778 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:53.653532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:53.653512 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:07:58.731093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.731056 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:07:58.735375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.735355 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.836659 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.836625 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:07:58.865327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865296 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865354 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdkjm\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865464 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865585 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865634 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865657 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865827 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865783 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865899 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865878 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.865990 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.865965 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.967895 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967817 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.967895 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967878 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdkjm\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967901 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967934 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967953 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.967967 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968026 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968137 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968223 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968287 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968337 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968374 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968662 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968548 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.968844 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.968806 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.970514 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.970493 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:58.970605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:58.970539 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:59.025609 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.025544 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdkjm\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:59.025609 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.025560 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:59.049363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.049339 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:07:59.481025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.480991 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:07:59.484587 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:07:59.484559 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84cac97_525e_49e1_9006_f9b39c1d2439.slice/crio-d132a41cebf7fb5e795ba64cf3469dba47869642ada7b39a68daf491c775cb10 WatchSource:0}: Error finding container d132a41cebf7fb5e795ba64cf3469dba47869642ada7b39a68daf491c775cb10: Status 404 returned error can't find the container with id d132a41cebf7fb5e795ba64cf3469dba47869642ada7b39a68daf491c775cb10 Nov 03 19:07:59.487218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.487182 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:59.487306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.487254 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:59.487306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.487282 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:59.909697 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.909661 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" event={"ID":"b84cac97-525e-49e1-9006-f9b39c1d2439","Type":"ContainerStarted","Data":"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974"} Nov 03 19:07:59.910282 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:07:59.909724 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" event={"ID":"b84cac97-525e-49e1-9006-f9b39c1d2439","Type":"ContainerStarted","Data":"d132a41cebf7fb5e795ba64cf3469dba47869642ada7b39a68daf491c775cb10"} Nov 03 19:08:00.025407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:00.025358 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podStartSLOduration=2.025344672 podStartE2EDuration="2.025344672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:00.022632872 +0000 UTC m=+1361.433732996" watchObservedRunningTime="2026-11-03 19:08:00.025344672 +0000 UTC m=+1361.436444795" Nov 03 19:08:00.050090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:00.050061 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:08:00.051489 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:00.051463 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:00.051604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:00.051509 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:01.049947 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:01.049913 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:01.050372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:01.049965 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:01.938552 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:08:01.938517 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:08:02.049766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:02.049730 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:02.049938 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:02.049780 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:03.050322 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:03.050284 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:03.050793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:03.050339 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:04.050527 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:04.050490 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:04.050922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:04.050552 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:05.049705 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:05.049670 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:05.049886 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:05.049736 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:06.050602 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:06.050570 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:08:06.051069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:06.050629 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 19:08:07.152880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:07.152847 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:08:07.153767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:07.153688 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:08:07.154896 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:07.154875 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:08:07.788799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:07.788765 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:08:10.921366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:10.921207 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.078447 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.078402 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.084282 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.084241 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.118407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.118364 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.268416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.268287 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.426144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.425961 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.901393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.898613 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 19:08:11.945191 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.942799 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 19:08:11.949925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:11.945969 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:08:12.073507 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.073217 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:08:12.089292 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.089245 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:08:12.092934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.092908 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:08:12.119221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.119183 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:08:12.135144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.119699 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 03 19:08:12.160533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.120565 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:08:12.302144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.277219 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 19:08:12.302839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.302806 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:08:12.424310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.424229 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:08:12.441143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.437219 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:08:12.607256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.607214 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.649793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.649765 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:08:12.742704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:12.742625 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:08:28.360069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.360030 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:08:28.365499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.365478 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522411 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522373 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522423 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522586 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522705 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522687 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzkk\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522754 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522883 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522869 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.522958 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522943 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.523014 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.522981 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.523112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.523100 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.568478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.568448 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:08:28.625248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625113 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625223 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625336 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625634 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625642 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625678 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625732 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.625849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625795 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzkk\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626013 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625860 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626013 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.625923 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.626012 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.626384 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626526 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.626503 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.626821 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.626800 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.627861 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.627830 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.628255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.628233 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.683414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.683378 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.683538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.683486 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzkk\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:28.978927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:28.978842 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:29.405431 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:29.405397 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:08:29.410282 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:08:29.410253 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c957be8_bd04_47f2_9089_f309ace6ff11.slice/crio-ae289e69a39b5f637c4e28a29f2af3bd8d312b9601c31e14ae52936a93799653 WatchSource:0}: Error finding container ae289e69a39b5f637c4e28a29f2af3bd8d312b9601c31e14ae52936a93799653: Status 404 returned error can't find the container with id ae289e69a39b5f637c4e28a29f2af3bd8d312b9601c31e14ae52936a93799653 Nov 03 19:08:29.412712 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:29.412673 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:29.412799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:29.412760 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:29.412871 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:29.412801 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:30.360955 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:30.360910 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" event={"ID":"6c957be8-bd04-47f2-9089-f309ace6ff11","Type":"ContainerStarted","Data":"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d"} Nov 03 19:08:30.361178 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:30.360975 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" event={"ID":"6c957be8-bd04-47f2-9089-f309ace6ff11","Type":"ContainerStarted","Data":"ae289e69a39b5f637c4e28a29f2af3bd8d312b9601c31e14ae52936a93799653"} Nov 03 19:08:30.521639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:30.521571 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podStartSLOduration=2.521551986 podStartE2EDuration="2.521551986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:30.520989848 +0000 UTC m=+1391.932090000" watchObservedRunningTime="2026-11-03 19:08:30.521551986 +0000 UTC m=+1391.932652111" Nov 03 19:08:30.981477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:30.981442 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:31.007300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:31.007255 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:31.007518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:31.007335 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:31.938071 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:08:31.938033 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:08:31.996142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:31.995473 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:31.996142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:31.995535 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:32.980929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:32.980458 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:32.980929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:32.980531 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:34.059055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:34.059004 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:34.060538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:34.060493 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:34.985143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:34.984518 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:34.985143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:34.984600 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:35.095255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.095209 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 19:08:35.278310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.269207 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.389095 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.364575 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 19:08:35.493283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.445487 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.578283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.578237 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.710425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.710382 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:08:35.776146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.750257 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.844511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.835215 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.844511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.835327 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 19:08:35.844511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.839830 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.899262 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.899218 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.983409 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.983288 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:35.983409 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:35.983350 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:36.072144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.071201 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.111200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.108233 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.111200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.109617 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:08:36.272265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.272219 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.308201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.308166 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:08:36.367884 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.367775 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.368536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.368509 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:08:36.461261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.461215 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.470586 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.467328 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:08:36.555143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.550741 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.623293 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.623194 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.649145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.623853 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:08:36.781183 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.775404 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:08:36.802145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.786330 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:08:36.900612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.816308 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.841688 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.841739 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.855252 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.908537 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.903713 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.931176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:36.881287 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.031540 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.031497 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.031732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.031577 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:37.058310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.058276 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:37.065150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.058553 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.088396 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.123476 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.169085 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.169167 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.169218 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.169256 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:08:37.178199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.169289 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:08:37.276084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.276057 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:08:37.319685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.293657 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:08:37.319685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.304811 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.377631 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.377549 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.466206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.465409 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.843225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.624156 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.843225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.726972 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:08:37.843225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.785239 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.966149 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.961103 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:08:37.968711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.968676 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.983594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.983461 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.983594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.983519 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:37.991155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:37.984062 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:38.155202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.110643 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:08:38.170171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.170109 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:38.348148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.326448 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:38.348148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.326902 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:08:38.348148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.329031 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:38.378091 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.378048 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 19:08:39.014145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.998418 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 03 19:08:39.014145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.998513 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.057145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.042393 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 19:08:39.057145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.053225 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:39.057145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.053992 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 19:08:39.086467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:38.841201 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:39.179856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.168222 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.179856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.163562 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.179856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.092181 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.179856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.168533 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:39.346965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.343739 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:08:39.347297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.347273 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:08:39.376355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.370365 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.481141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.401386 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.503204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.499370 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.547825 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.547775 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 19:08:39.594377 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.566883 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:08:39.674316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.674232 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:08:39.676684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.676663 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:08:39.676919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.676900 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:08:39.705605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.705564 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:08:39.709099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.709079 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:08:39.733408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.714783 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:08:39.747950 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.747926 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:08:39.979374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.979297 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.979374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:39.979357 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 19:08:41.041929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:41.041897 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:41.042464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:41.042244 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:41.043820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:41.043802 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:08:45.598018 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:45.597982 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:08:45.598414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:45.598243 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" containerID="cri-o://6a867e895cd0787d8f32856d9c935b34ddb35e346d0fff6155035b66c31b3228" gracePeriod=30 Nov 03 19:08:50.767940 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.767908 5209 generic.go:356] "Generic (PLEG): container finished" podID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerID="6a867e895cd0787d8f32856d9c935b34ddb35e346d0fff6155035b66c31b3228" exitCode=0 Nov 03 19:08:50.768395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.767950 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" event={"ID":"55b4de9e-4778-4fe3-8bd6-b17690da67e6","Type":"ContainerDied","Data":"6a867e895cd0787d8f32856d9c935b34ddb35e346d0fff6155035b66c31b3228"} Nov 03 19:08:50.886793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.886766 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:08:50.955776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.955747 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.955943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.955785 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.955943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.955925 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.955963 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956017 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956021 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:50.956071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956055 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956178 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956206 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956243 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\") pod \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\" (UID: \"55b4de9e-4778-4fe3-8bd6-b17690da67e6\") " Nov 03 19:08:50.956468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956366 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:50.956468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956439 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:50.956581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956562 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:50.956683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.956667 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:50.957380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.957350 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:50.957380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.957377 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:50.957532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.957392 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:50.957532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.957405 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:50.957532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.957421 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:50.958438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.958407 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:50.958542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.958487 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "kube-api-access-6p74n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:50.958876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.958857 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:50.958943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:50.958900 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token" pod "55b4de9e-4778-4fe3-8bd6-b17690da67e6" (UID: "55b4de9e-4778-4fe3-8bd6-b17690da67e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:51.059626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.059589 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p74n\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-kube-api-access-6p74n\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.059626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.059631 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.059871 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.059646 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.059871 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.059660 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b4de9e-4778-4fe3-8bd6-b17690da67e6-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.777682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.777654 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" Nov 03 19:08:51.778181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.777650 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4" event={"ID":"55b4de9e-4778-4fe3-8bd6-b17690da67e6","Type":"ContainerDied","Data":"547b4d7b9c480c95c00368993798302bd8a510b65930fb285a8a0bb7fd529e2b"} Nov 03 19:08:51.778181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.777803 5209 scope.go:122] "RemoveContainer" containerID="6a867e895cd0787d8f32856d9c935b34ddb35e346d0fff6155035b66c31b3228" Nov 03 19:08:51.982591 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:51.982540 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:08:52.028055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:52.027988 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z92v4"] Nov 03 19:08:52.699186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:08:52.699156 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" path="/var/lib/kubelet/pods/55b4de9e-4778-4fe3-8bd6-b17690da67e6/volumes" Nov 03 19:09:00.186032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.185981 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:09:00.187468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.187438 5209 cpu_manager.go:414] "containerMap: removing container" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" Nov 03 19:09:00.187600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.187479 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" Nov 03 19:09:00.187889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.187865 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b4de9e-4778-4fe3-8bd6-b17690da67e6" containerName="istio-proxy" Nov 03 19:09:00.194210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.194186 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.196613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.196583 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.196755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.196628 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.196755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.196698 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.196755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.196722 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.196932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.196898 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.197048 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.197030 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.197100 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.197086 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqlj\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.197351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.197323 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.197433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.197389 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.295215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.295180 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:09:00.298254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298229 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298272 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqlj\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298334 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298366 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298415 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298439 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298495 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298519 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298589 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298677 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.298989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298963 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.299453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.299427 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.299571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.299210 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.299571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.298998 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.301674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.301652 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.301755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.301679 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.356493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.356462 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.356649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.356529 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqlj\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.510085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.509995 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:00.725291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.725258 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:09:00.725491 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.725465 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" containerID="cri-o://b86838ebfd999d2fd0af9831b43256c7561109c47e54224981c4bdcbe2ba21ca" gracePeriod=30 Nov 03 19:09:00.937962 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.937930 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:09:00.943246 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:09:00.943219 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6e5660_3e8d_4c83_9287_058ea8bf9fa5.slice/crio-ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15 WatchSource:0}: Error finding container ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15: Status 404 returned error can't find the container with id ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15 Nov 03 19:09:00.945538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.945500 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:00.945656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.945588 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:00.945656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:00.945636 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:01.518551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.518478 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8878-ae49a6"] Nov 03 19:09:01.522956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.522932 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:01.572322 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.572295 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2f8z4\"" Nov 03 19:09:01.580958 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.580921 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8878-ae49a6"] Nov 03 19:09:01.618234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.618195 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57txd\" (UniqueName: \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\") pod \"test-curl-8878-ae49a6\" (UID: \"3c2bcdf3-c41f-42ea-abba-4472ba30eb11\") " pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:01.719304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.719271 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57txd\" (UniqueName: \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\") pod \"test-curl-8878-ae49a6\" (UID: \"3c2bcdf3-c41f-42ea-abba-4472ba30eb11\") " pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:01.802745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.802659 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57txd\" (UniqueName: \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\") pod \"test-curl-8878-ae49a6\" (UID: \"3c2bcdf3-c41f-42ea-abba-4472ba30eb11\") " pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:01.837555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.837522 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:01.863479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.863442 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" event={"ID":"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5","Type":"ContainerStarted","Data":"9b3e726b821500dce0d6197bebba000cac880fc581c53e7b862aa131d7bfa799"} Nov 03 19:09:01.863618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:01.863512 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" event={"ID":"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5","Type":"ContainerStarted","Data":"ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15"} Nov 03 19:09:01.948391 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:01.948328 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:09:02.008818 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.008743 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podStartSLOduration=2.008722226 podStartE2EDuration="2.008722226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:02.001019614 +0000 UTC m=+1423.412119738" watchObservedRunningTime="2026-11-03 19:09:02.008722226 +0000 UTC m=+1423.419822351" Nov 03 19:09:02.237007 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.236965 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8878-ae49a6"] Nov 03 19:09:02.242274 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:09:02.242239 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c2bcdf3_c41f_42ea_abba_4472ba30eb11.slice/crio-a96fc41adbc9a26ca5a4e28f5d49a8d370ff31407b74ee8c717b04ea3252636f WatchSource:0}: Error finding container a96fc41adbc9a26ca5a4e28f5d49a8d370ff31407b74ee8c717b04ea3252636f: Status 404 returned error can't find the container with id a96fc41adbc9a26ca5a4e28f5d49a8d370ff31407b74ee8c717b04ea3252636f Nov 03 19:09:02.244169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.244151 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:09:02.510822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.510731 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:02.512283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.512257 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:02.512398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.512302 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:02.871741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:02.871703 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8878-ae49a6" event={"ID":"3c2bcdf3-c41f-42ea-abba-4472ba30eb11","Type":"ContainerStarted","Data":"a96fc41adbc9a26ca5a4e28f5d49a8d370ff31407b74ee8c717b04ea3252636f"} Nov 03 19:09:03.511405 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:03.511365 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:03.511597 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:03.511431 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:04.511353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:04.511315 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:04.511837 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:04.511387 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:05.510857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:05.510812 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:05.511022 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:05.510887 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:05.921249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:05.921212 5209 generic.go:356] "Generic (PLEG): container finished" podID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerID="b86838ebfd999d2fd0af9831b43256c7561109c47e54224981c4bdcbe2ba21ca" exitCode=0 Nov 03 19:09:05.921616 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:05.921278 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" event={"ID":"5ce96116-b75b-4735-bdd9-e31a03f583fb","Type":"ContainerDied","Data":"b86838ebfd999d2fd0af9831b43256c7561109c47e54224981c4bdcbe2ba21ca"} Nov 03 19:09:05.922923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:05.922896 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8878-ae49a6" event={"ID":"3c2bcdf3-c41f-42ea-abba-4472ba30eb11","Type":"ContainerStarted","Data":"e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b"} Nov 03 19:09:06.035295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.035245 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8878-ae49a6" podStartSLOduration=1.570371467 podStartE2EDuration="5.03523001s" totalImagesPullingTime="3.464858543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:06.032430138 +0000 UTC m=+1427.443530284" watchObservedRunningTime="2026-11-03 19:09:06.03523001 +0000 UTC m=+1427.446330133" Nov 03 19:09:06.044311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.044291 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:09:06.111930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.111850 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.111930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.111889 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.111931 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112360 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112442 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112465 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112492 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.112677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112594 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.113281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112801 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:09:06.113281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112952 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:09:06.113281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.112983 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:09:06.113281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.113162 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:09:06.113281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.113104 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\") pod \"5ce96116-b75b-4735-bdd9-e31a03f583fb\" (UID: \"5ce96116-b75b-4735-bdd9-e31a03f583fb\") " Nov 03 19:09:06.113588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.113392 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114699 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114858 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114908 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114924 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114939 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.115142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.114951 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce96116-b75b-4735-bdd9-e31a03f583fb-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.115482 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.115362 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:09:06.115543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.115494 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "kube-api-access-nqqdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:09:06.115854 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.115835 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy" pod "5ce96116-b75b-4735-bdd9-e31a03f583fb" (UID: "5ce96116-b75b-4735-bdd9-e31a03f583fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:09:06.216859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.216815 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.216859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.216852 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqqdz\" (UniqueName: \"kubernetes.io/projected/5ce96116-b75b-4735-bdd9-e31a03f583fb-kube-api-access-nqqdz\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.216859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.216866 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.217182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.216874 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce96116-b75b-4735-bdd9-e31a03f583fb-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:06.511462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.511379 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:06.511462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.511452 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:06.935543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.935505 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" event={"ID":"5ce96116-b75b-4735-bdd9-e31a03f583fb","Type":"ContainerDied","Data":"6e4c2e03ba6403f6eba3723ee626391eca3b547a64acb1f11165b1d340361d8a"} Nov 03 19:09:06.936078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.935578 5209 scope.go:122] "RemoveContainer" containerID="b86838ebfd999d2fd0af9831b43256c7561109c47e54224981c4bdcbe2ba21ca" Nov 03 19:09:06.936078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:06.935523 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p" Nov 03 19:09:07.092466 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.092422 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:09:07.138350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.138311 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-99d5p"] Nov 03 19:09:07.510916 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.510874 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:09:07.511158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.510935 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 19:09:07.529782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.529744 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8878-ae49a6"] Nov 03 19:09:07.534009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.533976 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8878-ae49a6"] Nov 03 19:09:07.695943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.695903 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8878-aca6be"] Nov 03 19:09:07.696514 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696496 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" Nov 03 19:09:07.696514 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696513 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" Nov 03 19:09:07.696674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696538 5209 cpu_manager.go:414] "containerMap: removing container" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" containerName="test-curl-8878-ae49a6" Nov 03 19:09:07.696674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696543 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" containerName="test-curl-8878-ae49a6" Nov 03 19:09:07.696674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696631 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" containerName="istio-proxy" Nov 03 19:09:07.696674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.696644 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" containerName="test-curl-8878-ae49a6" Nov 03 19:09:07.699983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.699963 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:07.756527 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.756496 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8878-aca6be"] Nov 03 19:09:07.843728 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.843688 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pbl\" (UniqueName: \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\") pod \"test-curl-8878-aca6be\" (UID: \"3d3069aa-6823-4043-b401-b5c79948d897\") " pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:07.944785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.944744 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pbl\" (UniqueName: \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\") pod \"test-curl-8878-aca6be\" (UID: \"3d3069aa-6823-4043-b401-b5c79948d897\") " pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:07.945283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:07.944871 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8878-ae49a6" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" containerName="test-curl-8878-ae49a6" containerID="cri-o://e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b" gracePeriod=2 Nov 03 19:09:07.993435 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:07.993392 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:08.038755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.038723 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pbl\" (UniqueName: \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\") pod \"test-curl-8878-aca6be\" (UID: \"3d3069aa-6823-4043-b401-b5c79948d897\") " pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:08.313743 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.313662 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:08.514679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.514650 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:08.515158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.515130 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:08.515876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.515861 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:09:08.695454 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:09:08.695426 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d3069aa_6823_4043_b401_b5c79948d897.slice/crio-412d0e42f06b675c9c79266945cd2ab4f56667de3c7cd0c54368c1533346490c WatchSource:0}: Error finding container 412d0e42f06b675c9c79266945cd2ab4f56667de3c7cd0c54368c1533346490c: Status 404 returned error can't find the container with id 412d0e42f06b675c9c79266945cd2ab4f56667de3c7cd0c54368c1533346490c Nov 03 19:09:08.701188 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.701107 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce96116-b75b-4735-bdd9-e31a03f583fb" path="/var/lib/kubelet/pods/5ce96116-b75b-4735-bdd9-e31a03f583fb/volumes" Nov 03 19:09:08.724494 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:08.724468 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:08.730695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.730668 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8878-aca6be"] Nov 03 19:09:08.952803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.952704 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8878-aca6be" event={"ID":"3d3069aa-6823-4043-b401-b5c79948d897","Type":"ContainerStarted","Data":"0f90cb2987dcda9edb61f3c2a549947f9145fe0c738165dbd184f795f0868e50"} Nov 03 19:09:08.952803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:08.952758 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8878-aca6be" event={"ID":"3d3069aa-6823-4043-b401-b5c79948d897","Type":"ContainerStarted","Data":"412d0e42f06b675c9c79266945cd2ab4f56667de3c7cd0c54368c1533346490c"} Nov 03 19:09:10.194228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.194199 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:10.242237 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:10.242139 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:10.299456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.299415 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\" (UniqueName: \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\") pod \"3c2bcdf3-c41f-42ea-abba-4472ba30eb11\" (UID: \"3c2bcdf3-c41f-42ea-abba-4472ba30eb11\") " Nov 03 19:09:10.301504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.301473 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd" pod "3c2bcdf3-c41f-42ea-abba-4472ba30eb11" (UID: "3c2bcdf3-c41f-42ea-abba-4472ba30eb11"). InnerVolumeSpecName "kube-api-access-57txd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:09:10.401062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.401028 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57txd\" (UniqueName: \"kubernetes.io/projected/3c2bcdf3-c41f-42ea-abba-4472ba30eb11-kube-api-access-57txd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:10.605321 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.605265 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8878-aca6be" podStartSLOduration=3.60524879 podStartE2EDuration="3.60524879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:09.064440305 +0000 UTC m=+1430.475540445" watchObservedRunningTime="2026-11-03 19:09:10.60524879 +0000 UTC m=+1432.016348913" Nov 03 19:09:10.605860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.605668 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8878-aca6be"] Nov 03 19:09:10.611055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.611026 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8878-aca6be"] Nov 03 19:09:10.700209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.700177 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" path="/var/lib/kubelet/pods/3c2bcdf3-c41f-42ea-abba-4472ba30eb11/volumes" Nov 03 19:09:10.970318 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.970220 5209 generic.go:356] "Generic (PLEG): container finished" podID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" containerID="e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b" exitCode=137 Nov 03 19:09:10.970318 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.970308 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:10.970538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.970353 5209 scope.go:122] "RemoveContainer" containerID="e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b" Nov 03 19:09:10.970538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.970526 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8878-aca6be" podUID="3d3069aa-6823-4043-b401-b5c79948d897" containerName="test-curl-8878-aca6be" containerID="cri-o://0f90cb2987dcda9edb61f3c2a549947f9145fe0c738165dbd184f795f0868e50" gracePeriod=2 Nov 03 19:09:10.979731 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.979712 5209 scope.go:122] "RemoveContainer" containerID="e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b" Nov 03 19:09:10.980026 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:10.980003 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b\": container with ID starting with e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b not found: ID does not exist" containerID="e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b" Nov 03 19:09:10.980086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:10.980048 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b"} err="failed to get container status \"e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b\": rpc error: code = NotFound desc = could not find container \"e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b\": container with ID starting with e4aedf43c1e99e037632c412be0285afff528085501b778bc7b0900a425ec00b not found: ID does not exist" Nov 03 19:09:11.018173 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:11.018144 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:11.066063 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:11.066030 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:12.308102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.308068 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:09:12.309103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.309069 5209 cpu_manager.go:414] "containerMap: removing container" podUID="3d3069aa-6823-4043-b401-b5c79948d897" containerName="test-curl-8878-aca6be" Nov 03 19:09:12.309305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.309279 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3069aa-6823-4043-b401-b5c79948d897" containerName="test-curl-8878-aca6be" Nov 03 19:09:12.309519 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.309505 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3069aa-6823-4043-b401-b5c79948d897" containerName="test-curl-8878-aca6be" Nov 03 19:09:12.314860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.314836 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328545 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328515 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328560 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7slk\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328590 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328636 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328663 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328757 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328899 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.328989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328960 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.329108 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.328994 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.401170 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.397487 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e114ecf4-gw-openshift-default-dockercfg-hbm8z\"" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.436755 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.436812 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7slk\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.436843 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.436885 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.436912 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.437015 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.437212 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.437285 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.437318 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.439578 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.439811 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.440687 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.440935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.440830 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.441787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.441097 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.441787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.441553 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.458153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.457221 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:09:12.467012 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.466986 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.505687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.505650 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7slk\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.523166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.522611 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\") pod \"e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.540164 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:12.537755 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-ae49a6\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3c2bcdf3-c41f-42ea-abba-4472ba30eb11" pod="opendatahub/test-curl-8878-ae49a6" Nov 03 19:09:12.631956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.631915 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:12.990491 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:12.990464 5209 generic.go:356] "Generic (PLEG): container finished" podID="3d3069aa-6823-4043-b401-b5c79948d897" containerID="0f90cb2987dcda9edb61f3c2a549947f9145fe0c738165dbd184f795f0868e50" exitCode=137 Nov 03 19:09:13.082265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.082201 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:09:13.084878 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:09:13.084829 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7781a017_a247_4225_8663_29ef0a84789f.slice/crio-465527576e65a45ffbdd232e711a9169d2d3a07ff139df6aac19d469103b9b2c WatchSource:0}: Error finding container 465527576e65a45ffbdd232e711a9169d2d3a07ff139df6aac19d469103b9b2c: Status 404 returned error can't find the container with id 465527576e65a45ffbdd232e711a9169d2d3a07ff139df6aac19d469103b9b2c Nov 03 19:09:13.088037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.088001 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:13.088179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.088070 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:13.088179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.088115 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:13.184515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.184494 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:13.358367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.357011 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\" (UniqueName: \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\") pod \"3d3069aa-6823-4043-b401-b5c79948d897\" (UID: \"3d3069aa-6823-4043-b401-b5c79948d897\") " Nov 03 19:09:13.365763 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.365728 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl" pod "3d3069aa-6823-4043-b401-b5c79948d897" (UID: "3d3069aa-6823-4043-b401-b5c79948d897"). InnerVolumeSpecName "kube-api-access-l8pbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:09:13.471685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:13.471649 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8pbl\" (UniqueName: \"kubernetes.io/projected/3d3069aa-6823-4043-b401-b5c79948d897-kube-api-access-l8pbl\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:14.001512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.001459 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerStarted","Data":"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3"} Nov 03 19:09:14.001692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.001538 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerStarted","Data":"465527576e65a45ffbdd232e711a9169d2d3a07ff139df6aac19d469103b9b2c"} Nov 03 19:09:14.004870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.004846 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:14.005011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.004874 5209 scope.go:122] "RemoveContainer" containerID="0f90cb2987dcda9edb61f3c2a549947f9145fe0c738165dbd184f795f0868e50" Nov 03 19:09:14.162857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.162803 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podStartSLOduration=2.162790429 podStartE2EDuration="2.162790429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:14.160882697 +0000 UTC m=+1435.571982836" watchObservedRunningTime="2026-11-03 19:09:14.162790429 +0000 UTC m=+1435.573890552" Nov 03 19:09:14.208728 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:14.208683 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-aca6be\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3d3069aa-6823-4043-b401-b5c79948d897" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:14.554755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.554713 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:09:14.632823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.632789 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:14.638889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.638857 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:14.686807 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:14.686773 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8878-aca6be\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="3d3069aa-6823-4043-b401-b5c79948d897" pod="opendatahub/test-curl-8878-aca6be" Nov 03 19:09:14.700960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:14.700932 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3069aa-6823-4043-b401-b5c79948d897" path="/var/lib/kubelet/pods/3d3069aa-6823-4043-b401-b5c79948d897/volumes" Nov 03 19:09:15.013435 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:15.013402 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:15.014475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:15.014454 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:09:17.033269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.033219 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:09:17.071666 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.071612 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.072280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.053625 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:17.073145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.035532 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 19:09:17.283322 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.283217 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.318685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.318645 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 19:09:17.393257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.393214 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.752598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.749206 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:17.771149 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:17.758192 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.074284 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.075453 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.077393 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.078403 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.078477 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.078513 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:09:18.083201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.082280 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.084000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.083772 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:09:18.268320 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.268273 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Nov 03 19:09:18.316153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.303074 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.349200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.317815 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:09:18.443205 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.374195 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.455146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.451539 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:09:18.473284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.374759 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.642830 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.642636 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:09:18.784952 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.784909 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.809216 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.787540 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.812924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.812884 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:09:18.935748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.935649 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 19:09:18.980224 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:18.976347 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:19.083140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.066894 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:09:19.132202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.117270 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:09:19.132202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.117343 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:09:19.132202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.082226 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.132202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.083404 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.382271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.382225 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.408583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.385212 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.454144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.443492 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.852449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.852403 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 19:09:19.875194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.869141 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.012554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.981852 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:20.028206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.991103 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 19:09:20.028862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:19.994392 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:09:20.029622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.029599 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:09:20.030003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.029986 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:09:20.215456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.215350 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.413628 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.413585 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 19:09:20.414143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.413921 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.414558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.414538 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:09:20.433143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.426539 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g8wjl" podUID="90594851-ca2d-495b-bcbd-ed14c297e0b8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.15:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.466372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.460731 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 19:09:20.470328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.467227 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:20.489794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.489755 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.839159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.838209 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.867647 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.867597 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:20.867920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.867888 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.889247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.889195 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.911267 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:20.893214 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.039499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.039448 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.040819 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.039721 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:21.061181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.039750 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 19:09:21.216313 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.216222 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:09:21.229286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.229241 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.416530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.416477 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.425158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.422225 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.425158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.422286 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.425158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.422775 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.471642 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.471533 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 19:09:21.474034 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.471971 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:21.481959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.481827 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.817201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.806510 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.817201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.806817 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.817557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.817528 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.841145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.817695 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.877146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.842364 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.939145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.882589 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:09:21.974362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.896417 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.200816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.906257 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 19:09:22.201554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:21.980258 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.201809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.048634 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.204142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.049271 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:22.204687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.049316 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.239140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.140189 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.271164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.254456 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 03 19:09:22.422296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.422252 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.423012 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.422971 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.495565 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.495465 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.572478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.497043 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.882275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:22.882227 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.158796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.158706 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:09:23.201341 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.201295 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.202023 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.201987 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 19:09:23.215161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.215104 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.216472 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.215626 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:09:23.226920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.215657 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 19:09:23.256284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.256251 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:23.256847 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.256804 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.257052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.257031 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.450799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.450708 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:23.451283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.451247 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.470774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.470731 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 19:09:23.587182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.585081 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 19:09:23.599964 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.599920 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:23.605613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.605583 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:09:23.633185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.633155 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:09:23.712050 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.711976 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:09:23.760142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.758781 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:09:23.828063 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.828033 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:09:23.851547 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.851516 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:09:23.922624 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.922365 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:09:23.953256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.953225 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:09:23.965204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.965059 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:09:23.981902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:23.981873 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:09:26.618504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.618462 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:09:26.625479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.625441 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.668603 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.668571 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22v6z\" (UniqueName: \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.668803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.668609 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.678630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.678601 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:09:26.720217 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.720182 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e114ecf4\"" Nov 03 19:09:26.720413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.720253 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e114ecf4-dockercfg-w9tt8\"" Nov 03 19:09:26.769751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.769715 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22v6z\" (UniqueName: \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.769953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.769773 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.770498 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.770470 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.827962 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.827927 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22v6z\" (UniqueName: \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\") pod \"payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:26.943090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:26.943007 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:27.323549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:27.323474 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:09:27.327485 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:09:27.327452 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d5b418_0378_4984_a8e4_bb44b77f59d4.slice/crio-82a54d68ba8dd4863115c46fbb4c5defb7359b29926fa8cbf0d4a02dad8f6623 WatchSource:0}: Error finding container 82a54d68ba8dd4863115c46fbb4c5defb7359b29926fa8cbf0d4a02dad8f6623: Status 404 returned error can't find the container with id 82a54d68ba8dd4863115c46fbb4c5defb7359b29926fa8cbf0d4a02dad8f6623 Nov 03 19:09:27.734802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:27.734764 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" event={"ID":"a6d5b418-0378-4984-a8e4-bb44b77f59d4","Type":"ContainerStarted","Data":"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be"} Nov 03 19:09:27.735237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:27.734828 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" event={"ID":"a6d5b418-0378-4984-a8e4-bb44b77f59d4","Type":"ContainerStarted","Data":"82a54d68ba8dd4863115c46fbb4c5defb7359b29926fa8cbf0d4a02dad8f6623"} Nov 03 19:09:27.735237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:27.734867 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:09:27.844029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:27.843971 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" podStartSLOduration=1.843957118 podStartE2EDuration="1.843957118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:27.842449953 +0000 UTC m=+1449.253550094" watchObservedRunningTime="2026-11-03 19:09:27.843957118 +0000 UTC m=+1449.255057241" Nov 03 19:09:31.937249 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:09:31.937169 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:09:38.748987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:09:38.748959 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:10:01.938481 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:01.938445 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:10:02.494418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.494374 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:10:02.501156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.501100 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.550267 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.550220 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-46ccd045-serving-cert\"" Nov 03 19:10:02.576899 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.576866 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.577055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.576934 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqc2l\" (UniqueName: \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.577198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.577186 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.577283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.577269 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.602351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.602302 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:10:02.678266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.678228 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.678266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.678271 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.678496 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:02.678403 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-serving-cert: secret "maas-api-e2e-ait-46ccd045-serving-cert" not found Nov 03 19:10:02.678496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.678436 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.678496 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:02.678470 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls podName:3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86 nodeName:}" failed. No retries permitted until 2026-11-03 19:10:03.178448818 +0000 UTC m=+1484.589548942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls") pod "maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" (UID: "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86") : secret "maas-api-e2e-ait-46ccd045-serving-cert" not found Nov 03 19:10:02.678646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.678533 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqc2l\" (UniqueName: \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.679113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.679093 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.680873 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.680849 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:02.738933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:02.738895 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqc2l\" (UniqueName: \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:03.187633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:03.187598 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:03.189936 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:03.189911 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") pod \"maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:03.415720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:03.415684 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:03.793069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:03.792987 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:10:03.797745 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:10:03.797717 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a543dd3_48a9_4c0e_8a0b_5dd91f3c3c86.slice/crio-b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1 WatchSource:0}: Error finding container b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1: Status 404 returned error can't find the container with id b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1 Nov 03 19:10:04.063414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:04.063331 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" event={"ID":"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86","Type":"ContainerStarted","Data":"b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1"} Nov 03 19:10:05.071726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:05.071627 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" event={"ID":"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86","Type":"ContainerStarted","Data":"a9e9a1708b586aafc3d9ef4f755a74b7e0ad81ce2766a33454b40290f341aa6b"} Nov 03 19:10:05.072088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:05.071727 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:05.184638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:05.184575 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" podStartSLOduration=2.216311975 podStartE2EDuration="3.184557416s" totalImagesPullingTime="968.245441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:05.181024131 +0000 UTC m=+1486.592124254" watchObservedRunningTime="2026-11-03 19:10:05.184557416 +0000 UTC m=+1486.595657539" Nov 03 19:10:11.084111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:11.084079 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:10:11.179351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:11.179318 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:10:11.957152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:11.957104 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:10:11.961499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:11.961481 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.029332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.029290 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.029520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.029500 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.029694 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.029675 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.029779 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.029711 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlc6c\" (UniqueName: \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.058964 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.058937 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e114ecf4-serving-cert\"" Nov 03 19:10:12.065240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.065211 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:10:12.130785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.130748 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.130785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.130784 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlc6c\" (UniqueName: \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.131352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.130864 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.131352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.130953 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.131352 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:12.131096 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-serving-cert: secret "maas-api-e2e-migrate-e114ecf4-serving-cert" not found Nov 03 19:10:12.131352 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:12.131177 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls podName:bd7a8e29-1a9f-488a-b585-51ecbc369575 nodeName:}" failed. No retries permitted until 2026-11-03 19:10:12.631161512 +0000 UTC m=+1494.042261617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls") pod "maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" (UID: "bd7a8e29-1a9f-488a-b585-51ecbc369575") : secret "maas-api-e2e-migrate-e114ecf4-serving-cert" not found Nov 03 19:10:12.131515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.131388 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.133365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.133345 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.190094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.190072 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlc6c\" (UniqueName: \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.645625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.645589 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.648047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.648013 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") pod \"maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:12.877165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:12.877131 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:13.293518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:13.293438 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:10:13.298202 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:10:13.298172 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7a8e29_1a9f_488a_b585_51ecbc369575.slice/crio-6dd11439b159b4f3ec81576888101fc8accaaf4b11cf84292c4bfc1be18453a6 WatchSource:0}: Error finding container 6dd11439b159b4f3ec81576888101fc8accaaf4b11cf84292c4bfc1be18453a6: Status 404 returned error can't find the container with id 6dd11439b159b4f3ec81576888101fc8accaaf4b11cf84292c4bfc1be18453a6 Nov 03 19:10:14.161503 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:14.161465 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" event={"ID":"bd7a8e29-1a9f-488a-b585-51ecbc369575","Type":"ContainerStarted","Data":"6dd11439b159b4f3ec81576888101fc8accaaf4b11cf84292c4bfc1be18453a6"} Nov 03 19:10:15.169669 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:15.169636 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" event={"ID":"bd7a8e29-1a9f-488a-b585-51ecbc369575","Type":"ContainerStarted","Data":"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67"} Nov 03 19:10:15.170080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:15.169733 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:15.281546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:15.281493 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" podStartSLOduration=3.465966899 podStartE2EDuration="4.28147912s" totalImagesPullingTime="815.512221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:15.279716056 +0000 UTC m=+1496.690816179" watchObservedRunningTime="2026-11-03 19:10:15.28147912 +0000 UTC m=+1496.692579243" Nov 03 19:10:21.182468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:21.182362 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:10:31.438339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.438302 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:10:31.442618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.442598 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528358 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vmd\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528631 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528740 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528825 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528769 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528825 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528817 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528860 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.528941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528920 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.529042 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.528974 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.529113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.529091 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.544379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.544347 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:10:31.630510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630475 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630587 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vmd\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630618 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630680 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630694 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630723 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630745 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630788 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.630908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630837 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.631177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.630922 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.631242 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.631186 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.631242 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.631225 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.631353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.631252 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.631695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.631669 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.633006 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.632984 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.633300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.633284 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.689158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.689056 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.689270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.689183 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vmd\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\") pod \"maas-default-gateway-openshift-default-77f556d647-f7b7s\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.760445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:31.755899 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:31.943660 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:10:31.943629 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:10:32.308621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:32.308510 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:10:32.312726 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:10:32.312696 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bd9579_6f86_4854_8129_50996f0db9fa.slice/crio-e2851e8cc6f07bf75bc0aa84bef228e784a8fd49288f229d9c6e8a5582fe3fbd WatchSource:0}: Error finding container e2851e8cc6f07bf75bc0aa84bef228e784a8fd49288f229d9c6e8a5582fe3fbd: Status 404 returned error can't find the container with id e2851e8cc6f07bf75bc0aa84bef228e784a8fd49288f229d9c6e8a5582fe3fbd Nov 03 19:10:32.315168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:32.315138 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:32.315249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:32.315205 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:32.315249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:32.315235 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:33.310296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.310260 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" event={"ID":"b9bd9579-6f86-4854-8129-50996f0db9fa","Type":"ContainerStarted","Data":"0bab422f2f553187efc1d45145248f159175a944d32e109044f4dcdf84342ea0"} Nov 03 19:10:33.310687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.310316 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" event={"ID":"b9bd9579-6f86-4854-8129-50996f0db9fa","Type":"ContainerStarted","Data":"e2851e8cc6f07bf75bc0aa84bef228e784a8fd49288f229d9c6e8a5582fe3fbd"} Nov 03 19:10:33.525905 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.525834 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podStartSLOduration=2.525813456 podStartE2EDuration="2.525813456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:33.477653219 +0000 UTC m=+1514.888753347" watchObservedRunningTime="2026-11-03 19:10:33.525813456 +0000 UTC m=+1514.936913580" Nov 03 19:10:33.756760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.756727 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:33.757956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.757934 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:33.758012 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:33.757985 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:34.757504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:34.757472 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:34.757879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:34.757532 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:35.756915 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:35.756882 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:35.757090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:35.756941 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:36.757795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:36.757756 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:36.758309 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:36.757828 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:37.757324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:37.757280 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:37.757645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:37.757349 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:38.757079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:38.757044 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:38.757483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:38.757097 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:39.757243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.757195 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:10:39.757722 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.757295 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:10:39.807900 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.807812 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:10:39.812132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.812099 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.862381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.862345 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-46ccd045-openshift-default-dockercfg-87s4t\"" Nov 03 19:10:39.879061 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879020 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879250 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879081 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879250 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879216 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879264 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879335 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879395 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7c8r\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879472 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879641 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.879734 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.879692 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.916668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.916633 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:10:39.981196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981162 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981252 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981290 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981331 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981355 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981410 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7c8r\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981467 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981562 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981584 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981654 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.981992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.981905 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.982452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.982423 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.982620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.982452 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.982620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.982559 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.984306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.984282 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:39.984542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:39.984526 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:40.043597 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.043561 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7c8r\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:40.043877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.043855 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:40.128574 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.128535 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:40.565486 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.565393 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:10:40.582892 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:10:40.582859 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe12f59c_e89d_412f_a659_206f2d59f49a.slice/crio-6fb8e0eb9130d085436eab0989dff855f5b24c0a7608eb2ebdd293c2fea2fd55 WatchSource:0}: Error finding container 6fb8e0eb9130d085436eab0989dff855f5b24c0a7608eb2ebdd293c2fea2fd55: Status 404 returned error can't find the container with id 6fb8e0eb9130d085436eab0989dff855f5b24c0a7608eb2ebdd293c2fea2fd55 Nov 03 19:10:40.585659 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.585626 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:40.585764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.585702 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:40.585764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:40.585733 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:41.382638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.382603 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:41.383579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.383496 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:41.384233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.384213 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:10:41.411153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.411096 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" event={"ID":"be12f59c-e89d-412f-a659-206f2d59f49a","Type":"ContainerStarted","Data":"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc"} Nov 03 19:10:41.411345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.411175 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" event={"ID":"be12f59c-e89d-412f-a659-206f2d59f49a","Type":"ContainerStarted","Data":"6fb8e0eb9130d085436eab0989dff855f5b24c0a7608eb2ebdd293c2fea2fd55"} Nov 03 19:10:41.658013 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:41.657909 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podStartSLOduration=2.657888609 podStartE2EDuration="2.657888609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:41.655665359 +0000 UTC m=+1523.066765506" watchObservedRunningTime="2026-11-03 19:10:41.657888609 +0000 UTC m=+1523.068988733" Nov 03 19:10:42.129634 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:42.129597 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:42.131094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:42.131066 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:42.131094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:42.131132 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:43.129852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.129810 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:43.130407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.129862 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:43.428839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.428747 5209 generic.go:356] "Generic (PLEG): container finished" podID="7781a017-a247-4225-8663-29ef0a84789f" containerID="185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3" exitCode=0 Nov 03 19:10:43.428839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.428818 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerDied","Data":"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3"} Nov 03 19:10:43.429572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.429556 5209 scope.go:122] "RemoveContainer" containerID="185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3" Nov 03 19:10:43.430920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.430895 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:43.430996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.430956 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:43.430996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:43.430989 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:44.129190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.129152 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:44.129364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.129232 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:44.438646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.438571 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerStarted","Data":"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8"} Nov 03 19:10:44.635143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.633836 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:10:44.635143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.634003 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:44.635143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:44.634048 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:45.026144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:45.015220 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:10:45.130140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:45.129609 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:45.130140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:45.129656 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:45.633198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:45.632575 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:45.633198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:45.632636 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:46.129718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.129678 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:46.129897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.129757 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:46.561792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.561713 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:10:46.565225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.565200 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.632536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.632497 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:46.632718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.632574 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:46.740098 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740004 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740098 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740070 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740200 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740299 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740355 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740412 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740432 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740600 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.740954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.740725 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcfj\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.771757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.771283 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.841918 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842025 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842141 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842195 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842233 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842371 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842404 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.842527 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.843002 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.843135 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.843274 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.843718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.843620 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.844080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.844052 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.844487 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.844460 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcfj\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.846723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.846701 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.847329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.847308 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.925687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.925653 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:46.931244 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:46.929501 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcfj\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\") pod \"maas-default-gateway-openshift-default-77f556d647-8vfcl\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:47.129257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.129161 5209 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:10:47.129257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.129213 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 19:10:47.208431 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.208395 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:47.632847 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.632802 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:47.633051 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.632865 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:47.712766 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:10:47.712735 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda621c5bb_d630_4187_96b8_1799e249f0ed.slice/crio-e9a96ea3edfefff14c818d7dee76efc96c87544db623367de1cb9713a3a23497 WatchSource:0}: Error finding container e9a96ea3edfefff14c818d7dee76efc96c87544db623367de1cb9713a3a23497: Status 404 returned error can't find the container with id e9a96ea3edfefff14c818d7dee76efc96c87544db623367de1cb9713a3a23497 Nov 03 19:10:47.715488 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.715455 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:47.715583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.715523 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:47.715583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.715553 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:47.745979 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:47.745944 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:10:48.134739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.134709 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:48.135185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.135164 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:48.135831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.135813 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:10:48.472557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.472468 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" event={"ID":"a621c5bb-d630-4187-96b8-1799e249f0ed","Type":"ContainerStarted","Data":"d9b84ffa76726360feb3b558b835e7170c22060abc13023a7a75a977f0684ce8"} Nov 03 19:10:48.472557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.472523 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" event={"ID":"a621c5bb-d630-4187-96b8-1799e249f0ed","Type":"ContainerStarted","Data":"e9a96ea3edfefff14c818d7dee76efc96c87544db623367de1cb9713a3a23497"} Nov 03 19:10:48.633179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.633133 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:48.633390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:48.633186 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:49.210816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:49.210785 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:49.220144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:49.211467 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:49.220390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:49.220359 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:49.632715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:49.632669 5209 patch_prober.go:29] interesting pod/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:10:49.632933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:49.632734 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 19:10:50.210231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:50.210187 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:50.210441 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:50.210258 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:50.637798 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:50.637767 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:10:50.638881 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:50.638859 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:10:50.854418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:50.854369 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podStartSLOduration=4.854355166 podStartE2EDuration="4.854355166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:48.602689747 +0000 UTC m=+1530.013789871" watchObservedRunningTime="2026-11-03 19:10:50.854355166 +0000 UTC m=+1532.265455288" Nov 03 19:10:51.210526 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:51.210486 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:51.210710 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:51.210549 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:52.210557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:52.210521 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:52.211066 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:52.210580 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:52.319187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:52.319142 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:10:53.210161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:53.210114 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:53.210351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:53.210180 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:54.209737 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:54.209699 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:54.210187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:54.209772 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:55.209815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:55.209777 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8vfcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 19:10:55.210318 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:55.209841 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 19:10:56.660681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:56.660645 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:56.661112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:56.661054 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:10:56.661920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:10:56.661899 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:11:01.939348 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:11:01.939247 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:11:30.816338 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:30.816303 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:11:31.937219 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:11:31.937185 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:11:35.816909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:35.816854 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" podUID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" containerName="payload-processing" containerID="cri-o://44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be" gracePeriod=25 Nov 03 19:11:36.137673 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.137639 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:11:36.213030 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.213000 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\" (UniqueName: \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\") pod \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " Nov 03 19:11:36.213362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.213344 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\") pod \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\" (UID: \"a6d5b418-0378-4984-a8e4-bb44b77f59d4\") " Nov 03 19:11:36.213698 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.213675 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume" pod "a6d5b418-0378-4984-a8e4-bb44b77f59d4" (UID: "a6d5b418-0378-4984-a8e4-bb44b77f59d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:11:36.213891 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.213878 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d5b418-0378-4984-a8e4-bb44b77f59d4-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:36.215626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.215589 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z" pod "a6d5b418-0378-4984-a8e4-bb44b77f59d4" (UID: "a6d5b418-0378-4984-a8e4-bb44b77f59d4"). InnerVolumeSpecName "kube-api-access-22v6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:36.315307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.315276 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22v6z\" (UniqueName: \"kubernetes.io/projected/a6d5b418-0378-4984-a8e4-bb44b77f59d4-kube-api-access-22v6z\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:36.880200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.880164 5209 generic.go:356] "Generic (PLEG): container finished" podID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" containerID="44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be" exitCode=0 Nov 03 19:11:36.880604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.880230 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" Nov 03 19:11:36.880604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.880232 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" event={"ID":"a6d5b418-0378-4984-a8e4-bb44b77f59d4","Type":"ContainerDied","Data":"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be"} Nov 03 19:11:36.880604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.880320 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9" event={"ID":"a6d5b418-0378-4984-a8e4-bb44b77f59d4","Type":"ContainerDied","Data":"82a54d68ba8dd4863115c46fbb4c5defb7359b29926fa8cbf0d4a02dad8f6623"} Nov 03 19:11:36.880604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.880347 5209 scope.go:122] "RemoveContainer" containerID="44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be" Nov 03 19:11:36.891211 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.891181 5209 scope.go:122] "RemoveContainer" containerID="44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be" Nov 03 19:11:36.891512 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:11:36.891484 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be\": container with ID starting with 44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be not found: ID does not exist" containerID="44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be" Nov 03 19:11:36.891589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:36.891520 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be"} err="failed to get container status \"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be\": rpc error: code = NotFound desc = could not find container \"44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be\": container with ID starting with 44f3ccae06975cbdc24c42dd47c59e0bf12a94f89ab606890a10fb3ec5f420be not found: ID does not exist" Nov 03 19:11:37.042501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:37.042407 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:11:37.048297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:37.048268 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e114ecf4-6bf9dbb698-lpvq9"] Nov 03 19:11:38.701462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:38.701433 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" path="/var/lib/kubelet/pods/a6d5b418-0378-4984-a8e4-bb44b77f59d4/volumes" Nov 03 19:11:39.446474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:11:39.446442 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:12:01.630718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:01.630679 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:12:01.631191 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:01.630918 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" podUID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" containerName="maas-api" containerID="cri-o://a9e9a1708b586aafc3d9ef4f755a74b7e0ad81ce2766a33454b40290f341aa6b" gracePeriod=30 Nov 03 19:12:02.100159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.092985 5209 generic.go:356] "Generic (PLEG): container finished" podID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" containerID="a9e9a1708b586aafc3d9ef4f755a74b7e0ad81ce2766a33454b40290f341aa6b" exitCode=0 Nov 03 19:12:02.100159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.093103 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" event={"ID":"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86","Type":"ContainerDied","Data":"a9e9a1708b586aafc3d9ef4f755a74b7e0ad81ce2766a33454b40290f341aa6b"} Nov 03 19:12:02.100159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.093173 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" event={"ID":"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86","Type":"ContainerDied","Data":"b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1"} Nov 03 19:12:02.100159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.093197 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3a42a690429e7b9d795159e05f57e4ea02206877942dde63b1dca74b55ab8c1" Nov 03 19:12:02.123934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.123585 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:12:02.176756 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:02.175335 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:12:02.197529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.195992 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\") pod \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " Nov 03 19:12:02.197529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.196251 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\" (UniqueName: \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\") pod \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " Nov 03 19:12:02.197529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.196294 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\") pod \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " Nov 03 19:12:02.197529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.196446 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") pod \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\" (UID: \"3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86\") " Nov 03 19:12:02.199294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.199247 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca" pod "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" (UID: "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:02.200573 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.200544 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls" pod "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" (UID: "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:02.202706 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.202676 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs" pod "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" (UID: "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:02.202889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.202862 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l" pod "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" (UID: "3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86"). InnerVolumeSpecName "kube-api-access-hqc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:02.301842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.301662 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:02.301842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.301703 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqc2l\" (UniqueName: \"kubernetes.io/projected/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-kube-api-access-hqc2l\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:02.301842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.301720 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:02.301842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:02.301734 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:03.122827 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:03.116799 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl" Nov 03 19:12:03.277513 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:03.277454 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:12:03.328110 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:03.328050 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46ccd045-5bd7dbff59-j99nl"] Nov 03 19:12:04.649697 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:04.649662 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:12:04.650499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:04.650462 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" containerID="cri-o://3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc" gracePeriod=30 Nov 03 19:12:04.838251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:04.838214 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" path="/var/lib/kubelet/pods/3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86/volumes" Nov 03 19:12:09.389450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.389407 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:12:09.390014 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.389680 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" podUID="bd7a8e29-1a9f-488a-b585-51ecbc369575" containerName="maas-api" containerID="cri-o://60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67" gracePeriod=30 Nov 03 19:12:09.775332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.775300 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:12:09.835919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.832818 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\") pod \"bd7a8e29-1a9f-488a-b585-51ecbc369575\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " Nov 03 19:12:09.835919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.832951 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\" (UniqueName: \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\") pod \"bd7a8e29-1a9f-488a-b585-51ecbc369575\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " Nov 03 19:12:09.835919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.833263 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") pod \"bd7a8e29-1a9f-488a-b585-51ecbc369575\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " Nov 03 19:12:09.835919 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.833342 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\") pod \"bd7a8e29-1a9f-488a-b585-51ecbc369575\" (UID: \"bd7a8e29-1a9f-488a-b585-51ecbc369575\") " Nov 03 19:12:09.846846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.846797 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca" pod "bd7a8e29-1a9f-488a-b585-51ecbc369575" (UID: "bd7a8e29-1a9f-488a-b585-51ecbc369575"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:09.879912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.879650 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c" pod "bd7a8e29-1a9f-488a-b585-51ecbc369575" (UID: "bd7a8e29-1a9f-488a-b585-51ecbc369575"). InnerVolumeSpecName "kube-api-access-zlc6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:09.908531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.908486 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs" pod "bd7a8e29-1a9f-488a-b585-51ecbc369575" (UID: "bd7a8e29-1a9f-488a-b585-51ecbc369575"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:09.910902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.910712 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls" pod "bd7a8e29-1a9f-488a-b585-51ecbc369575" (UID: "bd7a8e29-1a9f-488a-b585-51ecbc369575"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:09.936187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.936005 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:09.936187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.936038 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7a8e29-1a9f-488a-b585-51ecbc369575-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:09.936187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.936055 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a8e29-1a9f-488a-b585-51ecbc369575-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:09.936187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.936069 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlc6c\" (UniqueName: \"kubernetes.io/projected/bd7a8e29-1a9f-488a-b585-51ecbc369575-kube-api-access-zlc6c\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:09.973663 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:09.973636 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:12:10.057806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.057767 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058063 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058329 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058461 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058695 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058838 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058825 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.058941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.058929 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.059052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.059039 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.059268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.059251 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\") pod \"be12f59c-e89d-412f-a659-206f2d59f49a\" (UID: \"be12f59c-e89d-412f-a659-206f2d59f49a\") " Nov 03 19:12:10.060505 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.060482 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:10.063297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.061172 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.063297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.062220 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.063297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.063203 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.063297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.063267 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.079144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.078764 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:12:10.079144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.078793 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "kube-api-access-j7c8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:10.083484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.083448 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:10.085020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.084991 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy" pod "be12f59c-e89d-412f-a659-206f2d59f49a" (UID: "be12f59c-e89d-412f-a659-206f2d59f49a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.161872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161837 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be12f59c-e89d-412f-a659-206f2d59f49a-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.161872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161869 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.161872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161879 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161890 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7c8r\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-kube-api-access-j7c8r\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161901 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be12f59c-e89d-412f-a659-206f2d59f49a-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161912 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161924 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161939 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be12f59c-e89d-412f-a659-206f2d59f49a-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.162137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.161948 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be12f59c-e89d-412f-a659-206f2d59f49a-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.182421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.182389 5209 generic.go:356] "Generic (PLEG): container finished" podID="be12f59c-e89d-412f-a659-206f2d59f49a" containerID="3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc" exitCode=0 Nov 03 19:12:10.182579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.182437 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" Nov 03 19:12:10.182579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.182467 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" event={"ID":"be12f59c-e89d-412f-a659-206f2d59f49a","Type":"ContainerDied","Data":"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc"} Nov 03 19:12:10.182579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.182523 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d" event={"ID":"be12f59c-e89d-412f-a659-206f2d59f49a","Type":"ContainerDied","Data":"6fb8e0eb9130d085436eab0989dff855f5b24c0a7608eb2ebdd293c2fea2fd55"} Nov 03 19:12:10.182579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.182545 5209 scope.go:122] "RemoveContainer" containerID="3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc" Nov 03 19:12:10.184155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.184104 5209 generic.go:356] "Generic (PLEG): container finished" podID="bd7a8e29-1a9f-488a-b585-51ecbc369575" containerID="60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67" exitCode=0 Nov 03 19:12:10.184295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.184179 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" Nov 03 19:12:10.184295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.184217 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" event={"ID":"bd7a8e29-1a9f-488a-b585-51ecbc369575","Type":"ContainerDied","Data":"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67"} Nov 03 19:12:10.184295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.184276 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n" event={"ID":"bd7a8e29-1a9f-488a-b585-51ecbc369575","Type":"ContainerDied","Data":"6dd11439b159b4f3ec81576888101fc8accaaf4b11cf84292c4bfc1be18453a6"} Nov 03 19:12:10.196940 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.196924 5209 scope.go:122] "RemoveContainer" containerID="3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc" Nov 03 19:12:10.197281 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:10.197261 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc\": container with ID starting with 3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc not found: ID does not exist" containerID="3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc" Nov 03 19:12:10.197351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.197288 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc"} err="failed to get container status \"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc\": rpc error: code = NotFound desc = could not find container \"3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc\": container with ID starting with 3621cdfa1eee8f8389c355f173c8f27621e5ab9ca47aabb66b36424ed73421fc not found: ID does not exist" Nov 03 19:12:10.197351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.197304 5209 scope.go:122] "RemoveContainer" containerID="60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67" Nov 03 19:12:10.207681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.207608 5209 scope.go:122] "RemoveContainer" containerID="60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67" Nov 03 19:12:10.208711 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:10.208677 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67\": container with ID starting with 60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67 not found: ID does not exist" containerID="60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67" Nov 03 19:12:10.208835 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.208708 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67"} err="failed to get container status \"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67\": rpc error: code = NotFound desc = could not find container \"60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67\": container with ID starting with 60549023cdd2ed02afec0d3648a684ccbc37acc578edf9c9daf0547ed6d2cc67 not found: ID does not exist" Nov 03 19:12:10.341848 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.341814 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:12:10.348386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.348359 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e114ecf4-5b78cd595b-rfm9n"] Nov 03 19:12:10.451342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.451252 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:12:10.455281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.455255 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-npc9d"] Nov 03 19:12:10.699436 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.699396 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7a8e29-1a9f-488a-b585-51ecbc369575" path="/var/lib/kubelet/pods/bd7a8e29-1a9f-488a-b585-51ecbc369575/volumes" Nov 03 19:12:10.699921 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:10.699901 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" path="/var/lib/kubelet/pods/be12f59c-e89d-412f-a659-206f2d59f49a/volumes" Nov 03 19:12:11.800715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:11.800673 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:12:11.801162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:11.800933 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" containerID="cri-o://510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8" gracePeriod=30 Nov 03 19:12:14.464259 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.464223 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:12:14.465114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465089 5209 cpu_manager.go:414] "containerMap: removing container" podUID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" containerName="maas-api" Nov 03 19:12:14.465314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465298 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" containerName="maas-api" Nov 03 19:12:14.465444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465426 5209 cpu_manager.go:414] "containerMap: removing container" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" Nov 03 19:12:14.465444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465442 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" Nov 03 19:12:14.465626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465486 5209 cpu_manager.go:414] "containerMap: removing container" podUID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" containerName="payload-processing" Nov 03 19:12:14.465626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465493 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" containerName="payload-processing" Nov 03 19:12:14.465626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465519 5209 cpu_manager.go:414] "containerMap: removing container" podUID="bd7a8e29-1a9f-488a-b585-51ecbc369575" containerName="maas-api" Nov 03 19:12:14.465626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465525 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7a8e29-1a9f-488a-b585-51ecbc369575" containerName="maas-api" Nov 03 19:12:14.465836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465697 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a543dd3-48a9-4c0e-8a0b-5dd91f3c3c86" containerName="maas-api" Nov 03 19:12:14.465836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465709 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="be12f59c-e89d-412f-a659-206f2d59f49a" containerName="istio-proxy" Nov 03 19:12:14.465836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465723 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d5b418-0378-4984-a8e4-bb44b77f59d4" containerName="payload-processing" Nov 03 19:12:14.465836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.465733 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7a8e29-1a9f-488a-b585-51ecbc369575" containerName="maas-api" Nov 03 19:12:14.503141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.495449 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.532382 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.529720 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:12:14.582509 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.582469 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5b012841\"" Nov 03 19:12:14.582700 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.582616 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5b012841-dockercfg-fjbps\"" Nov 03 19:12:14.622220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.622192 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:12:14.626376 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.626357 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.635114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.635089 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bv6r\" (UniqueName: \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.635237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.635204 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.680988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.680949 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:12:14.744218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.737690 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bv6r\" (UniqueName: \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.744218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.737758 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.744218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.737804 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggsvd\" (UniqueName: \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.744218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.737832 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.744218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.741731 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.811462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.811077 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bv6r\" (UniqueName: \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\") pod \"payload-processing-e2e-derive-5b012841-866ddc4c58-642db\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.835171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.816178 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:14.840478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.840436 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.840621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.840498 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggsvd\" (UniqueName: \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.841575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.841547 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.931090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.924589 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggsvd\" (UniqueName: \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\") pod \"payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:14.944289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:14.944232 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:15.211080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:15.211038 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:12:15.219508 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:12:15.219478 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9252ea2f_7ed2_4244_9f53_dd7b1430e90c.slice/crio-81f4e81b80fc9dce7ea0652a2df7a24c7aaf07d2a4c4880483393e23a9c96596 WatchSource:0}: Error finding container 81f4e81b80fc9dce7ea0652a2df7a24c7aaf07d2a4c4880483393e23a9c96596: Status 404 returned error can't find the container with id 81f4e81b80fc9dce7ea0652a2df7a24c7aaf07d2a4c4880483393e23a9c96596 Nov 03 19:12:15.234716 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:15.234691 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" event={"ID":"9252ea2f-7ed2-4244-9f53-dd7b1430e90c","Type":"ContainerStarted","Data":"81f4e81b80fc9dce7ea0652a2df7a24c7aaf07d2a4c4880483393e23a9c96596"} Nov 03 19:12:15.361274 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:15.361245 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:12:15.365613 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:12:15.365579 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b38e1a_c892_4ee5_8798_293a03c52067.slice/crio-ae7989f6a85323c5e7abb3b27648b623feb23279b3c37318b1ebc9cf740ebd08 WatchSource:0}: Error finding container ae7989f6a85323c5e7abb3b27648b623feb23279b3c37318b1ebc9cf740ebd08: Status 404 returned error can't find the container with id ae7989f6a85323c5e7abb3b27648b623feb23279b3c37318b1ebc9cf740ebd08 Nov 03 19:12:16.243653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.243617 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" event={"ID":"9252ea2f-7ed2-4244-9f53-dd7b1430e90c","Type":"ContainerStarted","Data":"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65"} Nov 03 19:12:16.244140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.243710 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:16.245421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.245400 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" event={"ID":"18b38e1a-c892-4ee5-8798-293a03c52067","Type":"ContainerStarted","Data":"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb"} Nov 03 19:12:16.245551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.245436 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" event={"ID":"18b38e1a-c892-4ee5-8798-293a03c52067","Type":"ContainerStarted","Data":"ae7989f6a85323c5e7abb3b27648b623feb23279b3c37318b1ebc9cf740ebd08"} Nov 03 19:12:16.245551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.245514 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:16.355384 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.355326 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" podStartSLOduration=2.355313254 podStartE2EDuration="2.355313254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:16.352845238 +0000 UTC m=+1617.763945377" watchObservedRunningTime="2026-11-03 19:12:16.355313254 +0000 UTC m=+1617.766413377" Nov 03 19:12:16.466021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:16.465959 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" podStartSLOduration=2.465939879 podStartE2EDuration="2.465939879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:16.461003533 +0000 UTC m=+1617.872103658" watchObservedRunningTime="2026-11-03 19:12:16.465939879 +0000 UTC m=+1617.877040005" Nov 03 19:12:17.129615 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.129587 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:12:17.255692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.255660 5209 generic.go:356] "Generic (PLEG): container finished" podID="7781a017-a247-4225-8663-29ef0a84789f" containerID="510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8" exitCode=0 Nov 03 19:12:17.256168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.255712 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" Nov 03 19:12:17.256168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.255737 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerDied","Data":"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8"} Nov 03 19:12:17.256168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.255797 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx" event={"ID":"7781a017-a247-4225-8663-29ef0a84789f","Type":"ContainerDied","Data":"465527576e65a45ffbdd232e711a9169d2d3a07ff139df6aac19d469103b9b2c"} Nov 03 19:12:17.256168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.255827 5209 scope.go:122] "RemoveContainer" containerID="510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8" Nov 03 19:12:17.267584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.267560 5209 scope.go:122] "RemoveContainer" containerID="185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3" Nov 03 19:12:17.279232 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.279210 5209 scope.go:122] "RemoveContainer" containerID="510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8" Nov 03 19:12:17.279571 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:17.279551 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8\": container with ID starting with 510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8 not found: ID does not exist" containerID="510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8" Nov 03 19:12:17.279671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.279587 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8"} err="failed to get container status \"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8\": rpc error: code = NotFound desc = could not find container \"510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8\": container with ID starting with 510b1aa52c65ea482d832785d4a3632b1cf57d572d0b2bf745e44411d8d518d8 not found: ID does not exist" Nov 03 19:12:17.279671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.279616 5209 scope.go:122] "RemoveContainer" containerID="185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3" Nov 03 19:12:17.279929 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:17.279908 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3\": container with ID starting with 185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3 not found: ID does not exist" containerID="185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3" Nov 03 19:12:17.279984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.279937 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3"} err="failed to get container status \"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3\": rpc error: code = NotFound desc = could not find container \"185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3\": container with ID starting with 185bd0fb7e8cc8678147557be47f00401bef3eb0ae1afd3f01d8c1bca2a77dd3 not found: ID does not exist" Nov 03 19:12:17.283419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.283396 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:12:17.284388 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.284368 5209 cpu_manager.go:414] "containerMap: removing container" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:17.284481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.284390 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:17.284603 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.284591 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:17.290497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.290479 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.295410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295166 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295244 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295321 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295351 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295483 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295532 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295570 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295582 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:17.295987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295677 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295711 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\") pod \"7781a017-a247-4225-8663-29ef0a84789f\" (UID: \"7781a017-a247-4225-8663-29ef0a84789f\") " Nov 03 19:12:17.295987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.295883 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:17.296153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296060 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:17.296400 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296377 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:17.296643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296624 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7781a017-a247-4225-8663-29ef0a84789f-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.296726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296648 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.296726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296664 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.296726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.296678 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.297327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.297250 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:17.300197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.299022 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:17.301711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.301671 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "kube-api-access-g7slk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:17.301811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.301757 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:17.301889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.301870 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo" pod "7781a017-a247-4225-8663-29ef0a84789f" (UID: "7781a017-a247-4225-8663-29ef0a84789f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:12:17.344050 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.343978 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:12:17.397964 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.397924 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398079 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2ps\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398205 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398329 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398383 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398417 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398454 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398483 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.398959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.398952 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.399395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.399058 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7781a017-a247-4225-8663-29ef0a84789f-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.399395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.399076 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.399395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.399092 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7781a017-a247-4225-8663-29ef0a84789f-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.399395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.399106 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7slk\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-kube-api-access-g7slk\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.399395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.399139 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7781a017-a247-4225-8663-29ef0a84789f-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:17.500556 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500501 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500605 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500658 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2ps\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500699 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500776 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500836 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500867 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500898 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.500983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.500924 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.501882 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.501845 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.502065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.502048 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.502443 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.502411 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.503015 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.502990 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.503612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.503590 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.503980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.503961 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.505323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.505304 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.559600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.559564 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.559881 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.559847 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2ps\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\") pod \"maas-default-gateway-openshift-default-77f556d647-xg2b8\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.638510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.638416 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:17.685722 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.685668 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:12:17.691524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:17.691489 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e114ecf4-gw-openshift-default-58b7c76569-zh9vx"] Nov 03 19:12:18.078157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.078115 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:12:18.082088 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:12:18.082062 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289f055e_b9a3_4ce7_8e59_db3cbf0fb1ee.slice/crio-524824e6254bdaec4bafd39dd1c54deccd24ecdb80dfb3ea8c8d3f000e78df1f WatchSource:0}: Error finding container 524824e6254bdaec4bafd39dd1c54deccd24ecdb80dfb3ea8c8d3f000e78df1f: Status 404 returned error can't find the container with id 524824e6254bdaec4bafd39dd1c54deccd24ecdb80dfb3ea8c8d3f000e78df1f Nov 03 19:12:18.085264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.085230 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:18.085398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.085296 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:18.085398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.085324 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:18.266655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.266601 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" event={"ID":"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee","Type":"ContainerStarted","Data":"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312"} Nov 03 19:12:18.267089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.266711 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" event={"ID":"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee","Type":"ContainerStarted","Data":"524824e6254bdaec4bafd39dd1c54deccd24ecdb80dfb3ea8c8d3f000e78df1f"} Nov 03 19:12:18.638733 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.638691 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:18.720033 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:18.719990 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7781a017-a247-4225-8663-29ef0a84789f" path="/var/lib/kubelet/pods/7781a017-a247-4225-8663-29ef0a84789f/volumes" Nov 03 19:12:19.640027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:19.639996 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:19.640423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:19.640047 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:20.639057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:20.639009 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:20.639276 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:20.639086 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:21.639425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:21.639389 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:21.639824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:21.639454 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:22.639777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:22.639734 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:22.640215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:22.639805 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:23.638893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:23.638856 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:23.639083 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:23.638920 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:24.638925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:24.638887 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:24.639330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:24.638952 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:25.639792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:25.639756 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xg2b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:12:25.640314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:25.639817 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 19:12:26.665405 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:26.665370 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:26.666035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:26.666015 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:26.666689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:26.666670 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:12:26.833918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:26.833865 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podStartSLOduration=9.833849711 podStartE2EDuration="9.833849711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:18.384652475 +0000 UTC m=+1619.795752609" watchObservedRunningTime="2026-11-03 19:12:26.833849711 +0000 UTC m=+1628.244949834" Nov 03 19:12:27.258696 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:27.258669 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:12:27.258876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:27.258747 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:12:29.008604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.008517 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:12:29.009480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.009453 5209 cpu_manager.go:414] "containerMap: removing container" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:29.009480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.009478 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:29.009652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.009605 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="7781a017-a247-4225-8663-29ef0a84789f" containerName="istio-proxy" Nov 03 19:12:29.013022 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.012999 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.063354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.063316 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 19:12:29.063354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.063330 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 19:12:29.063354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.063349 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 19:12:29.063653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.063574 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dqcwl\"" Nov 03 19:12:29.093085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.092785 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.092863 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093085 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.092908 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.093611 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jfcf\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.093689 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.093750 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.093978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.093784 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.122321 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.122288 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:12:29.195727 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.195694 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.195923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.195733 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.195923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.195891 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.196049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.195934 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.196049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.195983 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.196268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.196248 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jfcf\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.196371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.196319 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.196578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.196551 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.198136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.198104 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.198342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.198314 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.198429 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.198369 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.198581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.198566 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.254308 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.254275 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.254443 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.254311 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jfcf\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8glmx\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.329978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.329949 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:29.791276 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:12:29.791242 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262e4b92_2faa_4dea_8a7a_0804d0aba4d1.slice/crio-a373c3812b10d201857c44fd0dc0ff6f88e7e4b20cbee7a7a3c2c8c300bf0fc7 WatchSource:0}: Error finding container a373c3812b10d201857c44fd0dc0ff6f88e7e4b20cbee7a7a3c2c8c300bf0fc7: Status 404 returned error can't find the container with id a373c3812b10d201857c44fd0dc0ff6f88e7e4b20cbee7a7a3c2c8c300bf0fc7 Nov 03 19:12:29.836863 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.836827 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:12:29.900081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:29.900046 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:12:30.453932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:30.453890 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" event={"ID":"262e4b92-2faa-4dea-8a7a-0804d0aba4d1","Type":"ContainerStarted","Data":"a373c3812b10d201857c44fd0dc0ff6f88e7e4b20cbee7a7a3c2c8c300bf0fc7"} Nov 03 19:12:31.971917 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:12:31.971835 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:12:33.947717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:33.947682 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:33.948112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:33.947801 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:34.534281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:34.534239 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" event={"ID":"262e4b92-2faa-4dea-8a7a-0804d0aba4d1","Type":"ContainerStarted","Data":"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876"} Nov 03 19:12:34.534459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:34.534339 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:12:34.748617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:34.748566 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" podStartSLOduration=2.59550081 podStartE2EDuration="6.748553473s" totalImagesPullingTime="4.153052663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:34.698709378 +0000 UTC m=+1636.109809523" watchObservedRunningTime="2026-11-03 19:12:34.748553473 +0000 UTC m=+1636.159653596" Nov 03 19:12:35.543173 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:35.543137 5209 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8glmx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:12:35.543551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:35.543186 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:12:38.543071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:12:38.543029 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:13:01.937716 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:13:01.937680 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:13:12.122144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:12.121239 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:12.227144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:12.222335 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:12.227144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:12.226192 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:12.525388 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:12.525275 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:13:12.525661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:12.525615 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:15.885456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:15.885416 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:13:15.885939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:15.885674 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" containerID="cri-o://45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe" gracePeriod=30 Nov 03 19:13:20.904831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:20.904792 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:42828->10.133.0.61:15021: read: connection reset by peer" start-of-body= Nov 03 19:13:20.905280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:20.904879 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:42828->10.133.0.61:15021: read: connection reset by peer" Nov 03 19:13:21.199090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.199064 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:13:21.311374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311343 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311415 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311499 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311532 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311644 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311664 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:21.311745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311674 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311702 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311952 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311780 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.311952 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311799 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:21.311952 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.311817 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\") pod \"96737262-e6e0-4294-a311-04f0f0475bae\" (UID: \"96737262-e6e0-4294-a311-04f0f0475bae\") " Nov 03 19:13:21.312114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312006 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:21.312114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312034 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:21.312305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312281 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:21.312563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312542 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.312563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312564 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.312701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312572 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96737262-e6e0-4294-a311-04f0f0475bae-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.312701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312580 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.312701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.312589 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.313857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.313832 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:21.313937 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.313922 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:21.314195 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.314172 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:21.314281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.314248 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7" pod "96737262-e6e0-4294-a311-04f0f0475bae" (UID: "96737262-e6e0-4294-a311-04f0f0475bae"). InnerVolumeSpecName "kube-api-access-fcfk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:21.414299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.414268 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96737262-e6e0-4294-a311-04f0f0475bae-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.414299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.414296 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96737262-e6e0-4294-a311-04f0f0475bae-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.414299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.414306 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:21.414531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:21.414316 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcfk7\" (UniqueName: \"kubernetes.io/projected/96737262-e6e0-4294-a311-04f0f0475bae-kube-api-access-fcfk7\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:22.001856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.001813 5209 generic.go:356] "Generic (PLEG): container finished" podID="96737262-e6e0-4294-a311-04f0f0475bae" containerID="45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe" exitCode=0 Nov 03 19:13:22.001856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.001848 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" Nov 03 19:13:22.002378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.001909 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" event={"ID":"96737262-e6e0-4294-a311-04f0f0475bae","Type":"ContainerDied","Data":"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe"} Nov 03 19:13:22.002378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.001958 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr" event={"ID":"96737262-e6e0-4294-a311-04f0f0475bae","Type":"ContainerDied","Data":"0e87aad575885aeb716962b32a638fc11aeac10e04a644fdc57df91a70b1cc16"} Nov 03 19:13:22.002378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.001989 5209 scope.go:122] "RemoveContainer" containerID="45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe" Nov 03 19:13:22.014151 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.013941 5209 scope.go:122] "RemoveContainer" containerID="45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe" Nov 03 19:13:22.014237 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:13:22.014218 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe\": container with ID starting with 45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe not found: ID does not exist" containerID="45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe" Nov 03 19:13:22.014287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.014245 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe"} err="failed to get container status \"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe\": rpc error: code = NotFound desc = could not find container \"45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe\": container with ID starting with 45d6f61ca90d01c5a0e8a9bb6b69d46855f69bcb550e45d5a7e8ce2b541deabe not found: ID does not exist" Nov 03 19:13:22.210984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.210949 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:13:22.257263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.257193 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-9ksmr"] Nov 03 19:13:22.701810 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:22.701778 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96737262-e6e0-4294-a311-04f0f0475bae" path="/var/lib/kubelet/pods/96737262-e6e0-4294-a311-04f0f0475bae/volumes" Nov 03 19:13:31.942854 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:13:31.942814 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:13:46.740730 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:46.740692 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:13:50.429143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:50.428203 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:51.179145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:51.178213 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:51.303146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:13:51.298257 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:14:01.939370 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:01.939280 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:14:18.962993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:18.962962 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:14:19.061802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:19.061768 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:14:21.505943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.505904 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:14:21.506575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.506560 5209 cpu_manager.go:414] "containerMap: removing container" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" Nov 03 19:14:21.506618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.506575 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" Nov 03 19:14:21.506689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.506678 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="96737262-e6e0-4294-a311-04f0f0475bae" containerName="istio-proxy" Nov 03 19:14:21.509464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.509447 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.568270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.568240 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt26h\" (UniqueName: \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.568415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.568365 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.568459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.568411 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.568459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.568438 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.607592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.607560 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-08baeb-serving-cert\"" Nov 03 19:14:21.613632 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.613604 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:14:21.669655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.669612 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt26h\" (UniqueName: \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.669834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.669699 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.669834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.669720 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.669834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.669738 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.670003 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:21.669835 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-serving-cert: secret "maas-api-e2e-api-a-08baeb-serving-cert" not found Nov 03 19:14:21.670003 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:21.669910 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls podName:284b815e-43dc-476b-a473-874262fca5a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:14:22.169890439 +0000 UTC m=+1743.580990547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls") pod "maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" (UID: "284b815e-43dc-476b-a473-874262fca5a3") : secret "maas-api-e2e-api-a-08baeb-serving-cert" not found Nov 03 19:14:21.670392 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.670370 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.672360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.672335 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:21.727520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:21.727492 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt26h\" (UniqueName: \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:22.178038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:22.177998 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:22.181164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:22.181134 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") pod \"maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:22.421822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:22.421789 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:22.827729 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:22.827699 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:14:22.831821 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:14:22.831787 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284b815e_43dc_476b_a473_874262fca5a3.slice/crio-9258788fcf36e721215f58f96ba6e464b53646c30c6babea9f7e06f8c7f0d9f1 WatchSource:0}: Error finding container 9258788fcf36e721215f58f96ba6e464b53646c30c6babea9f7e06f8c7f0d9f1: Status 404 returned error can't find the container with id 9258788fcf36e721215f58f96ba6e464b53646c30c6babea9f7e06f8c7f0d9f1 Nov 03 19:14:22.833619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:22.833601 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:14:23.650193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:23.650081 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" event={"ID":"284b815e-43dc-476b-a473-874262fca5a3","Type":"ContainerStarted","Data":"9258788fcf36e721215f58f96ba6e464b53646c30c6babea9f7e06f8c7f0d9f1"} Nov 03 19:14:23.968346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:23.968199 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" podUID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" containerName="payload-processing" containerID="cri-o://1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65" gracePeriod=25 Nov 03 19:14:24.062573 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.062532 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" podUID="18b38e1a-c892-4ee5-8798-293a03c52067" containerName="payload-pre-processing" containerID="cri-o://325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb" gracePeriod=25 Nov 03 19:14:24.158550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.158506 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:14:24.372966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.372926 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:14:24.435412 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.435387 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\" (UniqueName: \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\") pod \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " Nov 03 19:14:24.435559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.435543 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\") pod \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\" (UID: \"9252ea2f-7ed2-4244-9f53-dd7b1430e90c\") " Nov 03 19:14:24.435904 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.435885 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume" pod "9252ea2f-7ed2-4244-9f53-dd7b1430e90c" (UID: "9252ea2f-7ed2-4244-9f53-dd7b1430e90c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:24.436135 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.436107 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:24.437720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.437699 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r" pod "9252ea2f-7ed2-4244-9f53-dd7b1430e90c" (UID: "9252ea2f-7ed2-4244-9f53-dd7b1430e90c"). InnerVolumeSpecName "kube-api-access-5bv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:24.438091 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.438076 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:14:24.537186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.537095 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\" (UniqueName: \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\") pod \"18b38e1a-c892-4ee5-8798-293a03c52067\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " Nov 03 19:14:24.537344 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.537274 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\") pod \"18b38e1a-c892-4ee5-8798-293a03c52067\" (UID: \"18b38e1a-c892-4ee5-8798-293a03c52067\") " Nov 03 19:14:24.537610 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.537588 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume" pod "18b38e1a-c892-4ee5-8798-293a03c52067" (UID: "18b38e1a-c892-4ee5-8798-293a03c52067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:24.537842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.537829 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b38e1a-c892-4ee5-8798-293a03c52067-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:24.537884 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.537845 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bv6r\" (UniqueName: \"kubernetes.io/projected/9252ea2f-7ed2-4244-9f53-dd7b1430e90c-kube-api-access-5bv6r\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:24.539093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.539072 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd" pod "18b38e1a-c892-4ee5-8798-293a03c52067" (UID: "18b38e1a-c892-4ee5-8798-293a03c52067"). InnerVolumeSpecName "kube-api-access-ggsvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:24.639591 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.639539 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggsvd\" (UniqueName: \"kubernetes.io/projected/18b38e1a-c892-4ee5-8798-293a03c52067-kube-api-access-ggsvd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:24.658358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.658325 5209 generic.go:356] "Generic (PLEG): container finished" podID="18b38e1a-c892-4ee5-8798-293a03c52067" containerID="325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb" exitCode=0 Nov 03 19:14:24.658497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.658377 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" Nov 03 19:14:24.658497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.658411 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" event={"ID":"18b38e1a-c892-4ee5-8798-293a03c52067","Type":"ContainerDied","Data":"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb"} Nov 03 19:14:24.658497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.658463 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc" event={"ID":"18b38e1a-c892-4ee5-8798-293a03c52067","Type":"ContainerDied","Data":"ae7989f6a85323c5e7abb3b27648b623feb23279b3c37318b1ebc9cf740ebd08"} Nov 03 19:14:24.658497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.658489 5209 scope.go:122] "RemoveContainer" containerID="325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb" Nov 03 19:14:24.660350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.660322 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" event={"ID":"284b815e-43dc-476b-a473-874262fca5a3","Type":"ContainerStarted","Data":"daf00fb2dcf8230b0f6bec0fe1c4c758f6c2124cfb95048c90d637295e1f4655"} Nov 03 19:14:24.660465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.660377 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:24.661683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.661660 5209 generic.go:356] "Generic (PLEG): container finished" podID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" containerID="1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65" exitCode=0 Nov 03 19:14:24.661789 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.661700 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" Nov 03 19:14:24.661834 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.661701 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" event={"ID":"9252ea2f-7ed2-4244-9f53-dd7b1430e90c","Type":"ContainerDied","Data":"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65"} Nov 03 19:14:24.661877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.661827 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db" event={"ID":"9252ea2f-7ed2-4244-9f53-dd7b1430e90c","Type":"ContainerDied","Data":"81f4e81b80fc9dce7ea0652a2df7a24c7aaf07d2a4c4880483393e23a9c96596"} Nov 03 19:14:24.669959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.669941 5209 scope.go:122] "RemoveContainer" containerID="325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb" Nov 03 19:14:24.670284 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:24.670259 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb\": container with ID starting with 325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb not found: ID does not exist" containerID="325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb" Nov 03 19:14:24.670373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.670294 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb"} err="failed to get container status \"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb\": rpc error: code = NotFound desc = could not find container \"325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb\": container with ID starting with 325d271f6616018c71b2b395911bae8dc3199bae53b07eef5d11a8d78fe6e9fb not found: ID does not exist" Nov 03 19:14:24.670373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.670316 5209 scope.go:122] "RemoveContainer" containerID="1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65" Nov 03 19:14:24.679764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.679749 5209 scope.go:122] "RemoveContainer" containerID="1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65" Nov 03 19:14:24.680005 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:24.679990 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65\": container with ID starting with 1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65 not found: ID does not exist" containerID="1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65" Nov 03 19:14:24.680053 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.680010 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65"} err="failed to get container status \"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65\": rpc error: code = NotFound desc = could not find container \"1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65\": container with ID starting with 1b680066a10fa1f16a81be0f4d20eb985df9dcb9040829ad36bb5a075df20f65 not found: ID does not exist" Nov 03 19:14:24.813856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.813771 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:14:24.817761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.817733 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b012841-866ddc4c58-642db"] Nov 03 19:14:24.919736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.919699 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:14:24.925382 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:24.925350 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b012841-54cffb979f-g8kfc"] Nov 03 19:14:25.038144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:25.034729 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" podStartSLOduration=3.248450085 podStartE2EDuration="4.034708168s" totalImagesPullingTime="786.258083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:25.031553185 +0000 UTC m=+1746.442653327" watchObservedRunningTime="2026-11-03 19:14:25.034708168 +0000 UTC m=+1746.445808293" Nov 03 19:14:26.706619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:26.706574 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b38e1a-c892-4ee5-8798-293a03c52067" path="/var/lib/kubelet/pods/18b38e1a-c892-4ee5-8798-293a03c52067/volumes" Nov 03 19:14:26.707231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:26.707203 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" path="/var/lib/kubelet/pods/9252ea2f-7ed2-4244-9f53-dd7b1430e90c/volumes" Nov 03 19:14:30.677982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:30.677957 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:14:31.940957 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:14:31.940914 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:14:43.754735 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.754701 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:14:43.755430 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755412 5209 cpu_manager.go:414] "containerMap: removing container" podUID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" containerName="payload-processing" Nov 03 19:14:43.755483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755431 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" containerName="payload-processing" Nov 03 19:14:43.755483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755464 5209 cpu_manager.go:414] "containerMap: removing container" podUID="18b38e1a-c892-4ee5-8798-293a03c52067" containerName="payload-pre-processing" Nov 03 19:14:43.755483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755470 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b38e1a-c892-4ee5-8798-293a03c52067" containerName="payload-pre-processing" Nov 03 19:14:43.755595 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755574 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b38e1a-c892-4ee5-8798-293a03c52067" containerName="payload-pre-processing" Nov 03 19:14:43.755595 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.755587 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="9252ea2f-7ed2-4244-9f53-dd7b1430e90c" containerName="payload-processing" Nov 03 19:14:43.757943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.757920 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835268 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835328 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835398 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835432 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835491 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.835644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.835575 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5z9\" (UniqueName: \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.856414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.856380 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d86d1cd0\"/\"default-dockercfg-zsdq4\"" Nov 03 19:14:43.856630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.856437 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d86d1cd0\"/\"e2e-lifecycle-model-d86d1cd0-kserve-self-signed-certs\"" Nov 03 19:14:43.856630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.856437 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d86d1cd0\"/\"kube-root-ca.crt\"" Nov 03 19:14:43.856630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.856489 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d86d1cd0\"/\"openshift-service-ca.crt\"" Nov 03 19:14:43.860284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.860259 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:14:43.950086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950051 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5z9\" (UniqueName: \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950279 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950330 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950407 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950447 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950562 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950517 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950760 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.950879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950755 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.951008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.950981 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.953635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.953603 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:43.954779 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:43.954745 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:44.029136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.029042 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5z9\" (UniqueName: \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\") pod \"e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:44.070047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.069768 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:44.489169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.489142 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:14:44.805518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.805434 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" event={"ID":"fd1d57f6-7961-4c50-8656-4bb58d85db96","Type":"ContainerStarted","Data":"28055bc983ddbd0c4cc3c44f384eb9b1c9506c5beed88bddb02f37f8a50c796a"} Nov 03 19:14:44.805518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.805474 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" event={"ID":"fd1d57f6-7961-4c50-8656-4bb58d85db96","Type":"ContainerStarted","Data":"f0fc07553295208d8c1048db0eca9ba049fbe1de79ae227a05e0ca9e1a76994a"} Nov 03 19:14:44.918027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:44.917959 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" podStartSLOduration=1.917942332 podStartE2EDuration="1.917942332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:44.91485582 +0000 UTC m=+1766.325955954" watchObservedRunningTime="2026-11-03 19:14:44.917942332 +0000 UTC m=+1766.329042455" Nov 03 19:14:45.070196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:45.070083 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:46.603753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:46.603724 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:14:46.604256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:46.603929 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" containerID="cri-o://9b3e726b821500dce0d6197bebba000cac880fc581c53e7b862aa131d7bfa799" gracePeriod=30 Nov 03 19:14:51.859499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:51.859467 5209 generic.go:356] "Generic (PLEG): container finished" podID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerID="9b3e726b821500dce0d6197bebba000cac880fc581c53e7b862aa131d7bfa799" exitCode=0 Nov 03 19:14:51.859851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:51.859655 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" event={"ID":"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5","Type":"ContainerDied","Data":"9b3e726b821500dce0d6197bebba000cac880fc581c53e7b862aa131d7bfa799"} Nov 03 19:14:51.859851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:51.859709 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" event={"ID":"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5","Type":"ContainerDied","Data":"ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15"} Nov 03 19:14:51.859851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:51.859729 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed25de41258b0782d4ac9c8c0ccde4da37fceda727765287435d7790016eba15" Nov 03 19:14:51.871666 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:51.871612 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:14:52.005572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.005412 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.005755 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.005855 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.005946 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.005981 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006023 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:52.006160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006086 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006371 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:52.006660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006634 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006720 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.006784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.006767 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\") pod \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\" (UID: \"5a6e5660-3e8d-4c83-9287-058ea8bf9fa5\") " Nov 03 19:14:52.044433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.044281 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:52.044433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.044413 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:52.058446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.047773 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:52.058446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.047889 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:52.058446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.047982 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "kube-api-access-rhqlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:52.058446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.048066 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:52.092351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.092289 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket" pod "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" (UID: "5a6e5660-3e8d-4c83-9287-058ea8bf9fa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:52.109641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109602 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109644 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqlj\" (UniqueName: \"kubernetes.io/projected/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-kube-api-access-rhqlj\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109663 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109679 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109700 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109716 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109732 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109754 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.109918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.109769 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:52.866525 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:52.866495 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct" Nov 03 19:14:53.070827 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:53.070676 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:14:53.086782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:53.086752 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-2g5ct"] Nov 03 19:14:54.070075 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:54.070038 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:54.083799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:54.083771 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:14:54.699751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:54.699716 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" path="/var/lib/kubelet/pods/5a6e5660-3e8d-4c83-9287-058ea8bf9fa5/volumes" Nov 03 19:14:54.898957 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:14:54.898929 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:15:00.205357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.205305 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:00.206649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.206627 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" Nov 03 19:15:00.206737 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.206650 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" Nov 03 19:15:00.206924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.206905 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6e5660-3e8d-4c83-9287-058ea8bf9fa5" containerName="istio-proxy" Nov 03 19:15:00.210939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.210917 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:00.211071 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.211047 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.214301 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.214281 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:00.214403 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.214383 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.217063 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.217043 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.242069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.242050 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:00.245002 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.244981 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.245495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.245473 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:00.248761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.248744 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.316595 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.316563 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:00.323622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.323594 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:00.329272 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.329239 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:00.332607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332572 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.332785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332620 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.332785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332726 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.332785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332759 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.332982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332854 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6dw\" (UniqueName: \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.332982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.332945 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9cf\" (UniqueName: \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.333064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.333019 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlb8\" (UniqueName: \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.333204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.333184 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.333351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.333335 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.333413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.333371 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdd2v\" (UniqueName: \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.356795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.356764 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:15:00.361705 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.361676 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:00.386082 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.386050 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:00.434840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434806 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.434840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434841 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdd2v\" (UniqueName: \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.435073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434912 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.435073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434932 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.435073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434956 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.435073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.434980 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.435073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435041 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6dw\" (UniqueName: \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.435399 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435084 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9cf\" (UniqueName: \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.435399 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435201 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlb8\" (UniqueName: \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.435399 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435258 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.435673 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435651 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.435757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435728 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.435828 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435750 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.435973 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.435951 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.436064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.436048 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.494034 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.493976 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdd2v\" (UniqueName: \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.494034 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.493978 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6dw\" (UniqueName: \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.494212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.494068 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9cf\" (UniqueName: \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\") pod \"maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.494422 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.494406 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlb8\" (UniqueName: \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\") pod \"maas-api-key-cleanup-29895555-wtpkb\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.494800 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.494784 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.526999 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.526977 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:00.534394 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.534375 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:00.541315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.541281 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:00.575011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.574973 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:00.577778 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.577755 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:00.914171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.914141 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:00.918000 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:15:00.917967 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11df8c1e_cfa1_4de5_814b_2f4acae0308a.slice/crio-120c8c67c4015f6e20005772312ee3315f541e73fd0a2194c9011ef51d4dd33d WatchSource:0}: Error finding container 120c8c67c4015f6e20005772312ee3315f541e73fd0a2194c9011ef51d4dd33d: Status 404 returned error can't find the container with id 120c8c67c4015f6e20005772312ee3315f541e73fd0a2194c9011ef51d4dd33d Nov 03 19:15:00.930293 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.930262 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerStarted","Data":"120c8c67c4015f6e20005772312ee3315f541e73fd0a2194c9011ef51d4dd33d"} Nov 03 19:15:00.966739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.966710 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:00.970177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:00.970154 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:00.971091 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:15:00.971059 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c62ca4f_d060_43c8_877c_5128dcc53d73.slice/crio-03e0ee25c388b3d1b0816d46185032368bffe4022f06e00b94a32895ab7a211e WatchSource:0}: Error finding container 03e0ee25c388b3d1b0816d46185032368bffe4022f06e00b94a32895ab7a211e: Status 404 returned error can't find the container with id 03e0ee25c388b3d1b0816d46185032368bffe4022f06e00b94a32895ab7a211e Nov 03 19:15:00.971969 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:15:00.971926 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a3bba6_2fba_417d_ac01_a1f1448f1eeb.slice/crio-33e6fc92ddc91fc6b5fe70f24e116e198abef68a1e535684e0fe57ccc5b9a563 WatchSource:0}: Error finding container 33e6fc92ddc91fc6b5fe70f24e116e198abef68a1e535684e0fe57ccc5b9a563: Status 404 returned error can't find the container with id 33e6fc92ddc91fc6b5fe70f24e116e198abef68a1e535684e0fe57ccc5b9a563 Nov 03 19:15:01.018366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.018338 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:01.022089 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:15:01.022065 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752b3ff3_82d0_4e3f_aeb3_c9a4ff9bdd6c.slice/crio-6a42302b8fadc3e62af692d893132ecaa074504a79418206abe2e5dd897f7cbb WatchSource:0}: Error finding container 6a42302b8fadc3e62af692d893132ecaa074504a79418206abe2e5dd897f7cbb: Status 404 returned error can't find the container with id 6a42302b8fadc3e62af692d893132ecaa074504a79418206abe2e5dd897f7cbb Nov 03 19:15:01.044053 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.043960 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:01.055256 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:15:01.055232 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c365e2_01b4_4104_95ce_61baf6ef5df2.slice/crio-1d8d8d67e4fd927803b01f875dd3f30372a3d4b5077583aebc8f3baff51e202d WatchSource:0}: Error finding container 1d8d8d67e4fd927803b01f875dd3f30372a3d4b5077583aebc8f3baff51e202d: Status 404 returned error can't find the container with id 1d8d8d67e4fd927803b01f875dd3f30372a3d4b5077583aebc8f3baff51e202d Nov 03 19:15:01.664107 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.664076 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:15:01.664496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.664340 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" containerID="cri-o://60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a" gracePeriod=30 Nov 03 19:15:01.939518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.939480 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerStarted","Data":"ec872ebd3b69c7fd37e2b6d91b356c20ae338af6c1118e4fd0b55ac42c367592"} Nov 03 19:15:01.939719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.939536 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerStarted","Data":"33e6fc92ddc91fc6b5fe70f24e116e198abef68a1e535684e0fe57ccc5b9a563"} Nov 03 19:15:01.940943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.940912 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerStarted","Data":"7b686d935f877357f0476949646a49200ded60a167bc15f38fb10a8df83583b7"} Nov 03 19:15:01.942432 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.942410 5209 generic.go:356] "Generic (PLEG): container finished" podID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerID="deea541d272a24ee3e1987f1dc82d441e33f0bea4bcb5de83ada26b37ef7ab5d" exitCode=60 Nov 03 19:15:01.942533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.942454 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" event={"ID":"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c","Type":"ContainerDied","Data":"deea541d272a24ee3e1987f1dc82d441e33f0bea4bcb5de83ada26b37ef7ab5d"} Nov 03 19:15:01.942533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.942491 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" event={"ID":"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c","Type":"ContainerStarted","Data":"6a42302b8fadc3e62af692d893132ecaa074504a79418206abe2e5dd897f7cbb"} Nov 03 19:15:01.942685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.942671 5209 scope.go:122] "RemoveContainer" containerID="deea541d272a24ee3e1987f1dc82d441e33f0bea4bcb5de83ada26b37ef7ab5d" Nov 03 19:15:01.944107 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.944085 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerStarted","Data":"d90bbe1b8a8ae252f4ccc0cf44b98e5e9ee163e26623f3481cf4edb46941f471"} Nov 03 19:15:01.944235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.944142 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerStarted","Data":"1d8d8d67e4fd927803b01f875dd3f30372a3d4b5077583aebc8f3baff51e202d"} Nov 03 19:15:01.945772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.945668 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerStarted","Data":"2db6d0e9bba13d361b177d7395c70a390b254ff9b967ef1b1920c9a51fbabc4c"} Nov 03 19:15:01.945772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:01.945707 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerStarted","Data":"03e0ee25c388b3d1b0816d46185032368bffe4022f06e00b94a32895ab7a211e"} Nov 03 19:15:01.978653 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:01.978615 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:15:02.050429 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.049270 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" podStartSLOduration=2.049250775 podStartE2EDuration="2.049250775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.048009868 +0000 UTC m=+1783.459110003" watchObservedRunningTime="2026-11-03 19:15:02.049250775 +0000 UTC m=+1783.460350902" Nov 03 19:15:02.157375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.157265 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" podStartSLOduration=2.157246242 podStartE2EDuration="2.157246242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.156958432 +0000 UTC m=+1783.568058556" watchObservedRunningTime="2026-11-03 19:15:02.157246242 +0000 UTC m=+1783.568346366" Nov 03 19:15:02.376986 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.376920 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" podStartSLOduration=2.3769052139999998 podStartE2EDuration="2.376905214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.373986838 +0000 UTC m=+1783.785086964" watchObservedRunningTime="2026-11-03 19:15:02.376905214 +0000 UTC m=+1783.788005337" Nov 03 19:15:02.485996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.485887 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" podStartSLOduration=2.485873774 podStartE2EDuration="2.485873774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.482020137 +0000 UTC m=+1783.893120261" watchObservedRunningTime="2026-11-03 19:15:02.485873774 +0000 UTC m=+1783.896973897" Nov 03 19:15:02.955467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.955435 5209 generic.go:356] "Generic (PLEG): container finished" podID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerID="2db6d0e9bba13d361b177d7395c70a390b254ff9b967ef1b1920c9a51fbabc4c" exitCode=60 Nov 03 19:15:02.955971 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.955526 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerDied","Data":"2db6d0e9bba13d361b177d7395c70a390b254ff9b967ef1b1920c9a51fbabc4c"} Nov 03 19:15:02.955971 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.955879 5209 scope.go:122] "RemoveContainer" containerID="2db6d0e9bba13d361b177d7395c70a390b254ff9b967ef1b1920c9a51fbabc4c" Nov 03 19:15:02.957434 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.957414 5209 generic.go:356] "Generic (PLEG): container finished" podID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerID="ec872ebd3b69c7fd37e2b6d91b356c20ae338af6c1118e4fd0b55ac42c367592" exitCode=60 Nov 03 19:15:02.957562 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.957494 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerDied","Data":"ec872ebd3b69c7fd37e2b6d91b356c20ae338af6c1118e4fd0b55ac42c367592"} Nov 03 19:15:02.957767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.957749 5209 scope.go:122] "RemoveContainer" containerID="ec872ebd3b69c7fd37e2b6d91b356c20ae338af6c1118e4fd0b55ac42c367592" Nov 03 19:15:02.959213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.959180 5209 generic.go:356] "Generic (PLEG): container finished" podID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerID="7b686d935f877357f0476949646a49200ded60a167bc15f38fb10a8df83583b7" exitCode=60 Nov 03 19:15:02.959304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.959241 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerDied","Data":"7b686d935f877357f0476949646a49200ded60a167bc15f38fb10a8df83583b7"} Nov 03 19:15:02.959575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.959471 5209 scope.go:122] "RemoveContainer" containerID="7b686d935f877357f0476949646a49200ded60a167bc15f38fb10a8df83583b7" Nov 03 19:15:02.961355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.961338 5209 generic.go:356] "Generic (PLEG): container finished" podID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerID="65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425" exitCode=60 Nov 03 19:15:02.961428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.961408 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" event={"ID":"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c","Type":"ContainerDied","Data":"65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425"} Nov 03 19:15:02.961475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.961455 5209 scope.go:122] "RemoveContainer" containerID="deea541d272a24ee3e1987f1dc82d441e33f0bea4bcb5de83ada26b37ef7ab5d" Nov 03 19:15:02.961621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.961608 5209 scope.go:122] "RemoveContainer" containerID="65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425" Nov 03 19:15:02.961808 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:02.961789 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd_odh-ai-gateway-infra(752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" Nov 03 19:15:02.963147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.963113 5209 generic.go:356] "Generic (PLEG): container finished" podID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerID="d90bbe1b8a8ae252f4ccc0cf44b98e5e9ee163e26623f3481cf4edb46941f471" exitCode=60 Nov 03 19:15:02.963243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.963167 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerDied","Data":"d90bbe1b8a8ae252f4ccc0cf44b98e5e9ee163e26623f3481cf4edb46941f471"} Nov 03 19:15:02.963403 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:02.963389 5209 scope.go:122] "RemoveContainer" containerID="d90bbe1b8a8ae252f4ccc0cf44b98e5e9ee163e26623f3481cf4edb46941f471" Nov 03 19:15:03.973500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.973466 5209 generic.go:356] "Generic (PLEG): container finished" podID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerID="bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6" exitCode=60 Nov 03 19:15:03.973983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.973543 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerDied","Data":"bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6"} Nov 03 19:15:03.973983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.973602 5209 scope.go:122] "RemoveContainer" containerID="ec872ebd3b69c7fd37e2b6d91b356c20ae338af6c1118e4fd0b55ac42c367592" Nov 03 19:15:03.973983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.973943 5209 scope.go:122] "RemoveContainer" containerID="bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6" Nov 03 19:15:03.974376 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:03.974352 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg_odh-ai-gateway-infra(12a3bba6-2fba-417d-ac01-a1f1448f1eeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" Nov 03 19:15:03.975740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.975718 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerStarted","Data":"547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198"} Nov 03 19:15:03.978295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.978272 5209 scope.go:122] "RemoveContainer" containerID="65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425" Nov 03 19:15:03.978651 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:03.978628 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd_odh-ai-gateway-infra(752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" Nov 03 19:15:03.980701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.980670 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerStarted","Data":"9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497"} Nov 03 19:15:03.982625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:03.982596 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerStarted","Data":"2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a"} Nov 03 19:15:04.992618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.992583 5209 scope.go:122] "RemoveContainer" containerID="bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6" Nov 03 19:15:04.993051 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:04.992965 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg_odh-ai-gateway-infra(12a3bba6-2fba-417d-ac01-a1f1448f1eeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" Nov 03 19:15:04.994064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.994039 5209 generic.go:356] "Generic (PLEG): container finished" podID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerID="547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198" exitCode=60 Nov 03 19:15:04.994195 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.994149 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerDied","Data":"547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198"} Nov 03 19:15:04.994254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.994209 5209 scope.go:122] "RemoveContainer" containerID="7b686d935f877357f0476949646a49200ded60a167bc15f38fb10a8df83583b7" Nov 03 19:15:04.994418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.994391 5209 scope.go:122] "RemoveContainer" containerID="547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198" Nov 03 19:15:04.994753 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:04.994734 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk_odh-ai-gateway-infra(11df8c1e-cfa1-4de5-814b-2f4acae0308a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" Nov 03 19:15:04.995978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.995959 5209 generic.go:356] "Generic (PLEG): container finished" podID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerID="9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497" exitCode=60 Nov 03 19:15:04.996057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.996042 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerDied","Data":"9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497"} Nov 03 19:15:04.996286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.996268 5209 scope.go:122] "RemoveContainer" containerID="9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497" Nov 03 19:15:04.996566 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:04.996546 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-wtpkb_odh-ai-gateway-infra(f6c365e2-01b4-4104-95ce-61baf6ef5df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" Nov 03 19:15:04.997928 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.997909 5209 generic.go:356] "Generic (PLEG): container finished" podID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerID="2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a" exitCode=60 Nov 03 19:15:04.998009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.997976 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerDied","Data":"2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a"} Nov 03 19:15:04.998193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:04.998179 5209 scope.go:122] "RemoveContainer" containerID="2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a" Nov 03 19:15:04.998416 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:04.998401 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85_odh-ai-gateway-infra(6c62ca4f-d060-43c8-877c-5128dcc53d73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" Nov 03 19:15:05.006131 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:05.006098 5209 scope.go:122] "RemoveContainer" containerID="d90bbe1b8a8ae252f4ccc0cf44b98e5e9ee163e26623f3481cf4edb46941f471" Nov 03 19:15:05.017690 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:05.017666 5209 scope.go:122] "RemoveContainer" containerID="2db6d0e9bba13d361b177d7395c70a390b254ff9b967ef1b1920c9a51fbabc4c" Nov 03 19:15:06.008988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:06.008956 5209 scope.go:122] "RemoveContainer" containerID="9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497" Nov 03 19:15:06.009526 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:06.009387 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-wtpkb_odh-ai-gateway-infra(f6c365e2-01b4-4104-95ce-61baf6ef5df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" Nov 03 19:15:06.010862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:06.010839 5209 scope.go:122] "RemoveContainer" containerID="2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a" Nov 03 19:15:06.011264 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:06.011236 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85_odh-ai-gateway-infra(6c62ca4f-d060-43c8-877c-5128dcc53d73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" Nov 03 19:15:06.012671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:06.012648 5209 scope.go:122] "RemoveContainer" containerID="547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198" Nov 03 19:15:06.012937 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:06.012919 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk_odh-ai-gateway-infra(11df8c1e-cfa1-4de5-814b-2f4acae0308a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" Nov 03 19:15:06.963231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:06.963205 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:15:07.020044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.020017 5209 generic.go:356] "Generic (PLEG): container finished" podID="3a521666-3127-4542-95b0-0d51a945815d" containerID="60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a" exitCode=0 Nov 03 19:15:07.020500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.020064 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" Nov 03 19:15:07.020500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.020109 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" event={"ID":"3a521666-3127-4542-95b0-0d51a945815d","Type":"ContainerDied","Data":"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a"} Nov 03 19:15:07.020500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.020195 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr" event={"ID":"3a521666-3127-4542-95b0-0d51a945815d","Type":"ContainerDied","Data":"276b88a01aabfa2cc9fcbd2f087b8defc168bd882f169c1df7b1f80af963dc5e"} Nov 03 19:15:07.020500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.020219 5209 scope.go:122] "RemoveContainer" containerID="60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a" Nov 03 19:15:07.029964 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.029937 5209 scope.go:122] "RemoveContainer" containerID="60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a" Nov 03 19:15:07.030261 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:07.030243 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a\": container with ID starting with 60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a not found: ID does not exist" containerID="60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a" Nov 03 19:15:07.030334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.030269 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a"} err="failed to get container status \"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a\": rpc error: code = NotFound desc = could not find container \"60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a\": container with ID starting with 60a727e065686e598e0a9c2071db6d641a042de4453d932ae504d41dd4b5956a not found: ID does not exist" Nov 03 19:15:07.147863 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.147823 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.147863 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.147866 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.147990 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148018 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148055 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148084 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148099 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:07.148372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148221 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:07.148372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148228 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148343 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:07.148372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148362 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148624 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148444 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:07.148624 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148565 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\") pod \"3a521666-3127-4542-95b0-0d51a945815d\" (UID: \"3a521666-3127-4542-95b0-0d51a945815d\") " Nov 03 19:15:07.148789 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.148731 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:07.149275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.149252 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.149275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.149274 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.149511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.149283 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.149511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.149292 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.149511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.149301 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a521666-3127-4542-95b0-0d51a945815d-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.150458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.150433 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:07.150458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.150447 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:07.150681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.150663 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "kube-api-access-fdxqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:07.150749 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.150736 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy" pod "3a521666-3127-4542-95b0-0d51a945815d" (UID: "3a521666-3127-4542-95b0-0d51a945815d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:07.250685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.250607 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a521666-3127-4542-95b0-0d51a945815d-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.250685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.250641 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.250685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.250651 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a521666-3127-4542-95b0-0d51a945815d-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.250685 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.250664 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxqw\" (UniqueName: \"kubernetes.io/projected/3a521666-3127-4542-95b0-0d51a945815d-kube-api-access-fdxqw\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:07.437477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.437444 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:15:07.440876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:07.440851 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-62pcr"] Nov 03 19:15:08.701215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:08.701182 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a521666-3127-4542-95b0-0d51a945815d" path="/var/lib/kubelet/pods/3a521666-3127-4542-95b0-0d51a945815d/volumes" Nov 03 19:15:15.215165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:15.214492 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:15:15.215165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:15.215032 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" containerID="cri-o://3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d" gracePeriod=30 Nov 03 19:15:15.692256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:15.692215 5209 scope.go:122] "RemoveContainer" containerID="2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a" Nov 03 19:15:16.101660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:16.101629 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerStarted","Data":"735c6e1303551281fb27a5ace78c7e071e38fe3fd763cec24b07f2d2b029c130"} Nov 03 19:15:16.710143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:16.710050 5209 scope.go:122] "RemoveContainer" containerID="65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425" Nov 03 19:15:16.722624 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:16.712996 5209 scope.go:122] "RemoveContainer" containerID="9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497" Nov 03 19:15:16.725425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:16.725397 5209 scope.go:122] "RemoveContainer" containerID="547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198" Nov 03 19:15:16.921366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:16.920581 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:15:17.119171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:17.115658 5209 generic.go:356] "Generic (PLEG): container finished" podID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerID="735c6e1303551281fb27a5ace78c7e071e38fe3fd763cec24b07f2d2b029c130" exitCode=60 Nov 03 19:15:17.119171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:17.115791 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerDied","Data":"735c6e1303551281fb27a5ace78c7e071e38fe3fd763cec24b07f2d2b029c130"} Nov 03 19:15:17.119171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:17.115852 5209 scope.go:122] "RemoveContainer" containerID="2e5c520d68f45255a68158e9175d873e4201c52c7081e6c5d0a1f8fd46d68d2a" Nov 03 19:15:17.119171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:17.116599 5209 scope.go:122] "RemoveContainer" containerID="735c6e1303551281fb27a5ace78c7e071e38fe3fd763cec24b07f2d2b029c130" Nov 03 19:15:17.122934 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:17.122902 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85_odh-ai-gateway-infra(6c62ca4f-d060-43c8-877c-5128dcc53d73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" Nov 03 19:15:17.226651 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:17.226568 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:18.134997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.134963 5209 generic.go:356] "Generic (PLEG): container finished" podID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerID="dafc1761ff72eb59e1b0371bc1fdaddeb73dd4babafb80b185309856934e6e59" exitCode=60 Nov 03 19:15:18.135471 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.135040 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerDied","Data":"dafc1761ff72eb59e1b0371bc1fdaddeb73dd4babafb80b185309856934e6e59"} Nov 03 19:15:18.135471 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.135097 5209 scope.go:122] "RemoveContainer" containerID="547a9ffca83294b726d5a4233c8ff044ed4d8e8c187a06787202dfff7708b198" Nov 03 19:15:18.135634 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.135614 5209 scope.go:122] "RemoveContainer" containerID="dafc1761ff72eb59e1b0371bc1fdaddeb73dd4babafb80b185309856934e6e59" Nov 03 19:15:18.136003 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:18.135979 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk_odh-ai-gateway-infra(11df8c1e-cfa1-4de5-814b-2f4acae0308a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" Nov 03 19:15:18.138449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.138396 5209 generic.go:356] "Generic (PLEG): container finished" podID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerID="afb50dab2f9c93b6454d5139dc6114e70e7beb9548731f5d2656887bb7b86752" exitCode=60 Nov 03 19:15:18.138529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.138505 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" event={"ID":"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c","Type":"ContainerDied","Data":"afb50dab2f9c93b6454d5139dc6114e70e7beb9548731f5d2656887bb7b86752"} Nov 03 19:15:18.138889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.138868 5209 scope.go:122] "RemoveContainer" containerID="afb50dab2f9c93b6454d5139dc6114e70e7beb9548731f5d2656887bb7b86752" Nov 03 19:15:18.139216 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:18.139188 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd_odh-ai-gateway-infra(752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" Nov 03 19:15:18.140520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.140496 5209 generic.go:356] "Generic (PLEG): container finished" podID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerID="731cb0085aa899c8ee758fd847d1a8c86a97ed6fffe106be2923b8be64ef4bfa" exitCode=60 Nov 03 19:15:18.140635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.140583 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerDied","Data":"731cb0085aa899c8ee758fd847d1a8c86a97ed6fffe106be2923b8be64ef4bfa"} Nov 03 19:15:18.140994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.140976 5209 scope.go:122] "RemoveContainer" containerID="731cb0085aa899c8ee758fd847d1a8c86a97ed6fffe106be2923b8be64ef4bfa" Nov 03 19:15:18.141367 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:18.141303 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-wtpkb_odh-ai-gateway-infra(f6c365e2-01b4-4104-95ce-61baf6ef5df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" Nov 03 19:15:18.152277 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.152257 5209 scope.go:122] "RemoveContainer" containerID="65d72dafbe5c2b0b13e678bfbcf188848bf40de7c98618a180e8b528e3877425" Nov 03 19:15:18.168194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.168169 5209 scope.go:122] "RemoveContainer" containerID="9a9363f0086327bf425358681ffc678ef00ea588cfe698c48d7b88270b74e497" Nov 03 19:15:18.540692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.540666 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:18.654138 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.654092 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\" (UniqueName: \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\") pod \"6c62ca4f-d060-43c8-877c-5128dcc53d73\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " Nov 03 19:15:18.654472 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.654450 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\") pod \"6c62ca4f-d060-43c8-877c-5128dcc53d73\" (UID: \"6c62ca4f-d060-43c8-877c-5128dcc53d73\") " Nov 03 19:15:18.654757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.654737 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca" pod "6c62ca4f-d060-43c8-877c-5128dcc53d73" (UID: "6c62ca4f-d060-43c8-877c-5128dcc53d73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:18.655002 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.654985 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62ca4f-d060-43c8-877c-5128dcc53d73-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.656267 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.656241 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw" pod "6c62ca4f-d060-43c8-877c-5128dcc53d73" (UID: "6c62ca4f-d060-43c8-877c-5128dcc53d73"). InnerVolumeSpecName "kube-api-access-ng6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:18.692284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.692254 5209 scope.go:122] "RemoveContainer" containerID="bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6" Nov 03 19:15:18.757187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:18.757078 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6dw\" (UniqueName: \"kubernetes.io/projected/6c62ca4f-d060-43c8-877c-5128dcc53d73-kube-api-access-ng6dw\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.316150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:19.315666 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:19.405975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:19.389385 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:19.437474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:19.437435 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" event={"ID":"6c62ca4f-d060-43c8-877c-5128dcc53d73","Type":"ContainerDied","Data":"03e0ee25c388b3d1b0816d46185032368bffe4022f06e00b94a32895ab7a211e"} Nov 03 19:15:19.437720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:19.437706 5209 scope.go:122] "RemoveContainer" containerID="735c6e1303551281fb27a5ace78c7e071e38fe3fd763cec24b07f2d2b029c130" Nov 03 19:15:19.437970 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:19.437957 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85" Nov 03 19:15:20.028161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.024834 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:20.071483 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.070269 5209 scope.go:122] "RemoveContainer" containerID="9b3e726b821500dce0d6197bebba000cac880fc581c53e7b862aa131d7bfa799" Nov 03 19:15:20.328160 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.322560 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:20.389161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.388225 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\") pod \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " Nov 03 19:15:20.389161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.388276 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\" (UniqueName: \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\") pod \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\" (UID: \"11df8c1e-cfa1-4de5-814b-2f4acae0308a\") " Nov 03 19:15:20.390320 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.390280 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca" pod "11df8c1e-cfa1-4de5-814b-2f4acae0308a" (UID: "11df8c1e-cfa1-4de5-814b-2f4acae0308a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.503583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.468211 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v" pod "11df8c1e-cfa1-4de5-814b-2f4acae0308a" (UID: "11df8c1e-cfa1-4de5-814b-2f4acae0308a"). InnerVolumeSpecName "kube-api-access-rdd2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.503583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.493987 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11df8c1e-cfa1-4de5-814b-2f4acae0308a-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.503583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.494013 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdd2v\" (UniqueName: \"kubernetes.io/projected/11df8c1e-cfa1-4de5-814b-2f4acae0308a-kube-api-access-rdd2v\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.556213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:20.514350 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 19:15:21.087141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.082566 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 03 19:15:21.191534 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.189216 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.191534 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.191432 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 19:15:21.305340 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.297402 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.421147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.404211 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:21.421147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.404263 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895555-v5x85"] Nov 03 19:15:21.477937 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.477236 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.523180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.519357 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.523180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.520666 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.610264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.556055 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.610264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.557089 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:15:21.610264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.583921 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:21.610264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.597184 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:21.641040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.641000 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:21.647148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.645240 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerStarted","Data":"25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283"} Nov 03 19:15:21.647148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.645948 5209 scope.go:122] "RemoveContainer" containerID="25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283" Nov 03 19:15:21.647148 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:21.646623 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg_odh-ai-gateway-infra(12a3bba6-2fba-417d-ac01-a1f1448f1eeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" Nov 03 19:15:21.799216 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.796169 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" Nov 03 19:15:21.812929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.812881 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk" event={"ID":"11df8c1e-cfa1-4de5-814b-2f4acae0308a","Type":"ContainerDied","Data":"120c8c67c4015f6e20005772312ee3315f541e73fd0a2194c9011ef51d4dd33d"} Nov 03 19:15:21.813182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.813162 5209 scope.go:122] "RemoveContainer" containerID="dafc1761ff72eb59e1b0371bc1fdaddeb73dd4babafb80b185309856934e6e59" Nov 03 19:15:21.856140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.854815 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\" (UniqueName: \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\") pod \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " Nov 03 19:15:21.856140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.854919 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\") pod \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\" (UID: \"f6c365e2-01b4-4104-95ce-61baf6ef5df2\") " Nov 03 19:15:21.856140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.855062 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\") pod \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " Nov 03 19:15:21.857543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.857515 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca" pod "f6c365e2-01b4-4104-95ce-61baf6ef5df2" (UID: "f6c365e2-01b4-4104-95ce-61baf6ef5df2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.874336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.872022 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525" pod "752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" (UID: "752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c"). InnerVolumeSpecName "kube-api-access-jc525". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.884148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.881399 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\") pod \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\" (UID: \"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c\") " Nov 03 19:15:21.884148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.882472 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c365e2-01b4-4104-95ce-61baf6ef5df2-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.884148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.882497 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-kube-api-access-jc525\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.899166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.890344 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca" pod "752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" (UID: "752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.993878 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.989895 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:15:21.993878 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:21.990197 5209 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:22.009459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.007949 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.016750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.016722 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:15:22.117310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.113324 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.117310 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.113498 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.127587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.127508 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.127816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.127786 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.127959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.127943 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.128187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.128166 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.128454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.128438 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.128681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.128662 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.128866 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.128832 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\") pod \"6c957be8-bd04-47f2-9089-f309ace6ff11\" (UID: \"6c957be8-bd04-47f2-9089-f309ace6ff11\") " Nov 03 19:15:22.135312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.135190 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.135467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.135418 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.139221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.139192 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.139325 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.139234 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.145750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.145727 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.207742 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.206879 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.207742 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.206997 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "kube-api-access-zmzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.228221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.208093 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8" pod "f6c365e2-01b4-4104-95ce-61baf6ef5df2" (UID: "f6c365e2-01b4-4104-95ce-61baf6ef5df2"). InnerVolumeSpecName "kube-api-access-nnlb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.228221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.211862 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233348 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233389 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c957be8-bd04-47f2-9089-f309ace6ff11-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233406 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233420 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233437 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzkk\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-kube-api-access-zmzkk\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233454 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233469 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233483 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlb8\" (UniqueName: \"kubernetes.io/projected/f6c365e2-01b4-4104-95ce-61baf6ef5df2-kube-api-access-nnlb8\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.233497 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.238144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.234004 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo" pod "6c957be8-bd04-47f2-9089-f309ace6ff11" (UID: "6c957be8-bd04-47f2-9089-f309ace6ff11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.340945 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.340895 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c957be8-bd04-47f2-9089-f309ace6ff11-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.712668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.712633 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" path="/var/lib/kubelet/pods/6c62ca4f-d060-43c8-877c-5128dcc53d73/volumes" Nov 03 19:15:22.714359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.714337 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:22.714517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.714503 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895555-wv5sk"] Nov 03 19:15:22.818613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.818582 5209 generic.go:356] "Generic (PLEG): container finished" podID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerID="3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d" exitCode=0 Nov 03 19:15:22.818792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.818632 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" Nov 03 19:15:22.818792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.818661 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" event={"ID":"6c957be8-bd04-47f2-9089-f309ace6ff11","Type":"ContainerDied","Data":"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d"} Nov 03 19:15:22.818792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.818725 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978" event={"ID":"6c957be8-bd04-47f2-9089-f309ace6ff11","Type":"ContainerDied","Data":"ae289e69a39b5f637c4e28a29f2af3bd8d312b9601c31e14ae52936a93799653"} Nov 03 19:15:22.818792 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.818751 5209 scope.go:122] "RemoveContainer" containerID="3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d" Nov 03 19:15:22.821365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.821339 5209 generic.go:356] "Generic (PLEG): container finished" podID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerID="25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283" exitCode=60 Nov 03 19:15:22.821480 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.821424 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerDied","Data":"25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283"} Nov 03 19:15:22.821898 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.821876 5209 scope.go:122] "RemoveContainer" containerID="25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283" Nov 03 19:15:22.822288 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:22.822252 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg_odh-ai-gateway-infra(12a3bba6-2fba-417d-ac01-a1f1448f1eeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" Nov 03 19:15:22.824352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.824329 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" event={"ID":"752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c","Type":"ContainerDied","Data":"6a42302b8fadc3e62af692d893132ecaa074504a79418206abe2e5dd897f7cbb"} Nov 03 19:15:22.824448 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.824405 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd" Nov 03 19:15:22.826683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.826665 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" Nov 03 19:15:22.826766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.826694 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb" event={"ID":"f6c365e2-01b4-4104-95ce-61baf6ef5df2","Type":"ContainerDied","Data":"1d8d8d67e4fd927803b01f875dd3f30372a3d4b5077583aebc8f3baff51e202d"} Nov 03 19:15:22.832202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.832181 5209 scope.go:122] "RemoveContainer" containerID="3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d" Nov 03 19:15:22.832528 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:22.832503 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d\": container with ID starting with 3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d not found: ID does not exist" containerID="3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d" Nov 03 19:15:22.832623 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.832536 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d"} err="failed to get container status \"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d\": rpc error: code = NotFound desc = could not find container \"3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d\": container with ID starting with 3ea8fb3b7a53f0de6ea4c92303f34aeb7016f78aa426a2c887f4b18d0075520d not found: ID does not exist" Nov 03 19:15:22.832623 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.832558 5209 scope.go:122] "RemoveContainer" containerID="bd3a5186e07498ab88ccb8adcd55b40d58f4d156d1acce0ab0ca9909593dbbd6" Nov 03 19:15:22.848887 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.848860 5209 scope.go:122] "RemoveContainer" containerID="afb50dab2f9c93b6454d5139dc6114e70e7beb9548731f5d2656887bb7b86752" Nov 03 19:15:22.861641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.861622 5209 scope.go:122] "RemoveContainer" containerID="731cb0085aa899c8ee758fd847d1a8c86a97ed6fffe106be2923b8be64ef4bfa" Nov 03 19:15:22.982630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.982550 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:15:22.989271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:22.989243 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-fw978"] Nov 03 19:15:23.090911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:23.090875 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:23.096103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:23.096079 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895555-gwdpd"] Nov 03 19:15:23.199803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:23.199765 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:23.203422 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:23.203398 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-wtpkb"] Nov 03 19:15:23.451818 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:23.451784 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:24.084948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.084920 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:24.186996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.186903 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\" (UniqueName: \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\") pod \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " Nov 03 19:15:24.187188 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.187079 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\") pod \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\" (UID: \"12a3bba6-2fba-417d-ac01-a1f1448f1eeb\") " Nov 03 19:15:24.187380 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.187352 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca" pod "12a3bba6-2fba-417d-ac01-a1f1448f1eeb" (UID: "12a3bba6-2fba-417d-ac01-a1f1448f1eeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:24.187720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.187699 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:24.189155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.189098 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf" pod "12a3bba6-2fba-417d-ac01-a1f1448f1eeb" (UID: "12a3bba6-2fba-417d-ac01-a1f1448f1eeb"). InnerVolumeSpecName "kube-api-access-kh9cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:24.294182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.289898 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9cf\" (UniqueName: \"kubernetes.io/projected/12a3bba6-2fba-417d-ac01-a1f1448f1eeb-kube-api-access-kh9cf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:24.700613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.700580 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" path="/var/lib/kubelet/pods/11df8c1e-cfa1-4de5-814b-2f4acae0308a/volumes" Nov 03 19:15:24.701146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.701113 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" path="/var/lib/kubelet/pods/6c957be8-bd04-47f2-9089-f309ace6ff11/volumes" Nov 03 19:15:24.701581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.701564 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" path="/var/lib/kubelet/pods/752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c/volumes" Nov 03 19:15:24.702367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.702346 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" path="/var/lib/kubelet/pods/f6c365e2-01b4-4104-95ce-61baf6ef5df2/volumes" Nov 03 19:15:24.866652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.866404 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" event={"ID":"12a3bba6-2fba-417d-ac01-a1f1448f1eeb","Type":"ContainerDied","Data":"33e6fc92ddc91fc6b5fe70f24e116e198abef68a1e535684e0fe57ccc5b9a563"} Nov 03 19:15:24.866973 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.866925 5209 scope.go:122] "RemoveContainer" containerID="25a5a34684df093daeecfbf8ea642b92187f7f1ebe04a6e21ecc5fc8ff8b0283" Nov 03 19:15:24.867272 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:24.867251 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg" Nov 03 19:15:25.028934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:25.028853 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:25.074739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:25.074713 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d86d1cd0-29895555-m99rg"] Nov 03 19:15:26.701089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:26.701057 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" path="/var/lib/kubelet/pods/12a3bba6-2fba-417d-ac01-a1f1448f1eeb/volumes" Nov 03 19:15:31.531968 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:31.531889 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:15:31.532386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:31.532169 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" containerID="cri-o://d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f" gracePeriod=30 Nov 03 19:15:31.953678 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:31.953633 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:15:34.542106 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:34.542072 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:34.542502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:34.542152 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:36.813419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.813362 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:15:36.899209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899177 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899339 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899429 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899375 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899470 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899437 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:36.899470 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899449 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899536 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899601 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899581 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899615 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899693 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899773 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899728 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:36.899829 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899795 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\") pod \"ac42e280-2969-42ae-bbc3-68509c342310\" (UID: \"ac42e280-2969-42ae-bbc3-68509c342310\") " Nov 03 19:15:36.899982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.899954 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:36.900207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900180 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:36.900312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900287 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:36.900650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900625 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:36.900650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900652 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:36.900791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900665 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:36.900791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900677 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:36.900791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.900689 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e280-2969-42ae-bbc3-68509c342310-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:36.901614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.901593 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:36.901992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.901975 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "kube-api-access-6lvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:36.902241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.902219 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:36.902326 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.902264 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token" pod "ac42e280-2969-42ae-bbc3-68509c342310" (UID: "ac42e280-2969-42ae-bbc3-68509c342310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:36.974568 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.974528 5209 generic.go:356] "Generic (PLEG): container finished" podID="ac42e280-2969-42ae-bbc3-68509c342310" containerID="d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f" exitCode=0 Nov 03 19:15:36.974765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.974588 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" Nov 03 19:15:36.974765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.974615 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" event={"ID":"ac42e280-2969-42ae-bbc3-68509c342310","Type":"ContainerDied","Data":"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f"} Nov 03 19:15:36.974765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.974683 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th" event={"ID":"ac42e280-2969-42ae-bbc3-68509c342310","Type":"ContainerDied","Data":"23be1e57f48284a98458e8b6ba8ed4ad59db6a2c15d814501e6922e538c2a053"} Nov 03 19:15:36.974765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.974712 5209 scope.go:122] "RemoveContainer" containerID="d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f" Nov 03 19:15:36.987499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.987472 5209 scope.go:122] "RemoveContainer" containerID="d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f" Nov 03 19:15:36.987872 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:15:36.987844 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f\": container with ID starting with d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f not found: ID does not exist" containerID="d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f" Nov 03 19:15:36.987982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:36.987883 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f"} err="failed to get container status \"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f\": rpc error: code = NotFound desc = could not find container \"d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f\": container with ID starting with d3c8676d2e01e3811944ddeebc62ff1e29d54b1f7583f5adadbef74badd9698f not found: ID does not exist" Nov 03 19:15:37.002635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.002607 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e280-2969-42ae-bbc3-68509c342310-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:37.002754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.002640 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvm7\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-kube-api-access-6lvm7\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:37.002754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.002659 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e280-2969-42ae-bbc3-68509c342310-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:37.002754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.002674 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e280-2969-42ae-bbc3-68509c342310-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:37.131925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.131773 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:15:37.145723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:37.145691 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-524th"] Nov 03 19:15:38.700284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:15:38.700244 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac42e280-2969-42ae-bbc3-68509c342310" path="/var/lib/kubelet/pods/ac42e280-2969-42ae-bbc3-68509c342310/volumes" Nov 03 19:16:01.937607 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:16:01.937573 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:16:19.398843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:16:19.398805 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:16:22.279747 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:16:22.279713 5209 scope.go:122] "RemoveContainer" containerID="a9e9a1708b586aafc3d9ef4f755a74b7e0ad81ce2766a33454b40290f341aa6b" Nov 03 19:16:31.950776 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:16:31.950723 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:17:01.937691 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:01.937606 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:17:29.518018 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:29.517927 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:17:31.939228 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:31.939185 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:17:32.827764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:32.827732 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:17:32.827956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:32.827933 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" containerID="cri-o://93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24" gracePeriod=30 Nov 03 19:17:32.873568 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:32.873534 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:17:32.873761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:32.873740 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" containerID="cri-o://0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481" gracePeriod=30 Nov 03 19:17:34.352031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:34.351993 5209 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:17:34.352434 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:34.352054 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:17:38.218804 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.218779 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:17:38.222404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.222381 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:17:38.284278 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284249 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.284433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284311 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284370 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284433 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284411 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284453 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284490 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.284592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284510 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.284592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284548 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284600 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284660 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284671 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284691 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284734 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.284790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284766 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284825 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284872 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.284989 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285023 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285039 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.285099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285079 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.285415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285158 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\") pod \"376ad674-5c3d-49e7-9697-2dc47b219634\" (UID: \"376ad674-5c3d-49e7-9697-2dc47b219634\") " Nov 03 19:17:38.285415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285215 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\") pod \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\" (UID: \"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0\") " Nov 03 19:17:38.285415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285395 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.285714 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285692 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:38.285923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.285848 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286016 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286044 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286059 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286105 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286141 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286157 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286172 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.286902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286490 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.286902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.286554 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.287044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287023 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "kube-api-access-kssvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:38.287106 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287026 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:38.287708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287620 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:38.287708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287625 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:38.287708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287671 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.287708 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.287691 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:38.288106 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.288092 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy" pod "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" (UID: "0d6ce26c-0fb2-4bc9-b860-196ef1247bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:38.288601 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.288588 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:38.288683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.288671 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd" pod "376ad674-5c3d-49e7-9697-2dc47b219634" (UID: "376ad674-5c3d-49e7-9697-2dc47b219634"). InnerVolumeSpecName "kube-api-access-nt8jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387665 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387695 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8jd\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-kube-api-access-nt8jd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387709 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ad674-5c3d-49e7-9697-2dc47b219634-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387718 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387727 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ad674-5c3d-49e7-9697-2dc47b219634-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387735 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.387738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387743 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.388079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387752 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kssvd\" (UniqueName: \"kubernetes.io/projected/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-kube-api-access-kssvd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.388079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387762 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.388079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387770 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ad674-5c3d-49e7-9697-2dc47b219634-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.388079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.387778 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ad674-5c3d-49e7-9697-2dc47b219634-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.915840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.915806 5209 generic.go:356] "Generic (PLEG): container finished" podID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerID="93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24" exitCode=0 Nov 03 19:17:38.916025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.915867 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" Nov 03 19:17:38.916025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.915886 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" event={"ID":"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0","Type":"ContainerDied","Data":"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24"} Nov 03 19:17:38.916025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.915938 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9" event={"ID":"0d6ce26c-0fb2-4bc9-b860-196ef1247bc0","Type":"ContainerDied","Data":"d7cbc73255c7b0546312412681ab1ec14ef8781b32bcb2a28e536ecba0adbcd1"} Nov 03 19:17:38.916025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.915963 5209 scope.go:122] "RemoveContainer" containerID="93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24" Nov 03 19:17:38.918649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.917984 5209 generic.go:356] "Generic (PLEG): container finished" podID="376ad674-5c3d-49e7-9697-2dc47b219634" containerID="0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481" exitCode=0 Nov 03 19:17:38.918649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.918094 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" Nov 03 19:17:38.918914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.918886 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" event={"ID":"376ad674-5c3d-49e7-9697-2dc47b219634","Type":"ContainerDied","Data":"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481"} Nov 03 19:17:38.919044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.918943 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87" event={"ID":"376ad674-5c3d-49e7-9697-2dc47b219634","Type":"ContainerDied","Data":"bb4484b11cbc42bcf29fcea08dcf6dd92079f5687c711c7b74d161784f53b624"} Nov 03 19:17:38.927221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.927204 5209 scope.go:122] "RemoveContainer" containerID="93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24" Nov 03 19:17:38.927481 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:38.927463 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24\": container with ID starting with 93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24 not found: ID does not exist" containerID="93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24" Nov 03 19:17:38.927537 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.927488 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24"} err="failed to get container status \"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24\": rpc error: code = NotFound desc = could not find container \"93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24\": container with ID starting with 93b60ddb6f318e4314e840134ae4b52165da715185eec9ef327e238b63c20b24 not found: ID does not exist" Nov 03 19:17:38.927537 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.927505 5209 scope.go:122] "RemoveContainer" containerID="0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481" Nov 03 19:17:38.936367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.936352 5209 scope.go:122] "RemoveContainer" containerID="0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481" Nov 03 19:17:38.936591 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:38.936574 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481\": container with ID starting with 0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481 not found: ID does not exist" containerID="0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481" Nov 03 19:17:38.936637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:38.936598 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481"} err="failed to get container status \"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481\": rpc error: code = NotFound desc = could not find container \"0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481\": container with ID starting with 0389aa584a65abda0471e3054d78d0a5e3b1a75cdd415148a5f53d785ad7d481 not found: ID does not exist" Nov 03 19:17:39.073222 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:39.073140 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:17:39.119298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:39.119272 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-6z8w9"] Nov 03 19:17:39.180317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:39.180289 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:17:39.191405 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:39.191378 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-nlm87"] Nov 03 19:17:40.699861 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:40.699829 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" path="/var/lib/kubelet/pods/0d6ce26c-0fb2-4bc9-b860-196ef1247bc0/volumes" Nov 03 19:17:40.700428 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:40.700392 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" path="/var/lib/kubelet/pods/376ad674-5c3d-49e7-9697-2dc47b219634/volumes" Nov 03 19:17:47.922497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:47.922461 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:17:47.923009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:47.922649 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" containerID="cri-o://63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618" gracePeriod=30 Nov 03 19:17:48.030640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:48.030586 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:17:48.030926 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:48.030879 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" containerID="cri-o://985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150" gracePeriod=30 Nov 03 19:17:52.693929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:52.693891 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:17:52.694414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:52.693982 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:17:53.204418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.204345 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:17:53.278690 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278660 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278756 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278782 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278806 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278873 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278901 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.278994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278992 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.279096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.278992 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.279096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279019 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.279096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279050 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\") pod \"22dc0ad5-7047-4668-8563-eb5242bb96ec\" (UID: \"22dc0ad5-7047-4668-8563-eb5242bb96ec\") " Nov 03 19:17:53.279244 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279220 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.279812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279514 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.279812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279538 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.279812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279573 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.279812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279618 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.279812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.279645 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:53.281737 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.281715 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "kube-api-access-s297f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:53.281868 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.281849 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.281868 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.281858 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:53.281982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.281913 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo" pod "22dc0ad5-7047-4668-8563-eb5242bb96ec" (UID: "22dc0ad5-7047-4668-8563-eb5242bb96ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:53.303424 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.303388 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:17:53.380564 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380531 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.380724 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380625 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.380791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380732 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.380791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380765 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.380907 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380855 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.380983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380967 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.381048 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.380976 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.381134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381075 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.381134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381102 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.381261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381193 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\") pod \"361ba76d-8a0b-4e71-8240-525f86766a44\" (UID: \"361ba76d-8a0b-4e71-8240-525f86766a44\") " Nov 03 19:17:53.381872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381590 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:53.381872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381665 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.381872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381825 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.382144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.381877 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.382144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382079 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382101 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dc0ad5-7047-4668-8563-eb5242bb96ec-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382136 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382151 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382166 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382181 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382195 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382208 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361ba76d-8a0b-4e71-8240-525f86766a44-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382221 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382235 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s297f\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-kube-api-access-s297f\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382250 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.382378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.382263 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dc0ad5-7047-4668-8563-eb5242bb96ec-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.383689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.383652 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:53.384238 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.384215 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:53.384361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.384269 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:53.384361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.384325 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh" pod "361ba76d-8a0b-4e71-8240-525f86766a44" (UID: "361ba76d-8a0b-4e71-8240-525f86766a44"). InnerVolumeSpecName "kube-api-access-tmrsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:53.490668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.490589 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361ba76d-8a0b-4e71-8240-525f86766a44-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.490668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.490619 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.490668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.490629 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrsh\" (UniqueName: \"kubernetes.io/projected/361ba76d-8a0b-4e71-8240-525f86766a44-kube-api-access-tmrsh\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:53.490668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:53.490640 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361ba76d-8a0b-4e71-8240-525f86766a44-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:54.027362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.027330 5209 generic.go:356] "Generic (PLEG): container finished" podID="361ba76d-8a0b-4e71-8240-525f86766a44" containerID="985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150" exitCode=0 Nov 03 19:17:54.027788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.027373 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" Nov 03 19:17:54.027788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.027409 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" event={"ID":"361ba76d-8a0b-4e71-8240-525f86766a44","Type":"ContainerDied","Data":"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150"} Nov 03 19:17:54.027788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.027466 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x" event={"ID":"361ba76d-8a0b-4e71-8240-525f86766a44","Type":"ContainerDied","Data":"4d37034507e143b9de03b22816c5b0038fba632de8aa333e7e13c2e84830f362"} Nov 03 19:17:54.027788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.027497 5209 scope.go:122] "RemoveContainer" containerID="985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150" Nov 03 19:17:54.029093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.029066 5209 generic.go:356] "Generic (PLEG): container finished" podID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerID="63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618" exitCode=0 Nov 03 19:17:54.029220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.029110 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" Nov 03 19:17:54.029291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.029144 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" event={"ID":"22dc0ad5-7047-4668-8563-eb5242bb96ec","Type":"ContainerDied","Data":"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618"} Nov 03 19:17:54.029347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.029326 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn" event={"ID":"22dc0ad5-7047-4668-8563-eb5242bb96ec","Type":"ContainerDied","Data":"c90162b15fe70a1529b0280e311b3f337c855c39e371e692286dfd4f70e7063e"} Nov 03 19:17:54.039617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.039594 5209 scope.go:122] "RemoveContainer" containerID="985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150" Nov 03 19:17:54.039909 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:54.039885 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150\": container with ID starting with 985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150 not found: ID does not exist" containerID="985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150" Nov 03 19:17:54.039956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.039921 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150"} err="failed to get container status \"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150\": rpc error: code = NotFound desc = could not find container \"985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150\": container with ID starting with 985b3dfd64578d4aa9ed2c5f3a6b1422e071ad4924c8f8c4b0a324f1f5e33150 not found: ID does not exist" Nov 03 19:17:54.039956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.039947 5209 scope.go:122] "RemoveContainer" containerID="63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618" Nov 03 19:17:54.054165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.053317 5209 scope.go:122] "RemoveContainer" containerID="63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618" Nov 03 19:17:54.054165 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:17:54.053864 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618\": container with ID starting with 63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618 not found: ID does not exist" containerID="63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618" Nov 03 19:17:54.054165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.053893 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618"} err="failed to get container status \"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618\": rpc error: code = NotFound desc = could not find container \"63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618\": container with ID starting with 63a6ab6ea0fd0a9f7867e97c5c0d7da8401ac762057fd1aabff870d50d397618 not found: ID does not exist" Nov 03 19:17:54.183568 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.183529 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:17:54.230328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.230249 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-rnbkn"] Nov 03 19:17:54.291847 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.291813 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:17:54.297707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.297678 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-qxd9x"] Nov 03 19:17:54.699021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.698988 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" path="/var/lib/kubelet/pods/22dc0ad5-7047-4668-8563-eb5242bb96ec/volumes" Nov 03 19:17:54.699517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:17:54.699501 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" path="/var/lib/kubelet/pods/361ba76d-8a0b-4e71-8240-525f86766a44/volumes" Nov 03 19:18:01.937734 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:18:01.937697 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:18:03.028280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:03.028226 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:18:03.028754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:03.028485 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" containerID="cri-o://5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a" gracePeriod=30 Nov 03 19:18:03.074381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:03.074344 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:18:03.074635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:03.074609 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" containerID="cri-o://4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974" gracePeriod=30 Nov 03 19:18:04.919642 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:04.919607 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:18:04.920020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:04.919664 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:18:07.156823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:07.156786 5209 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:18:07.157457 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:07.157423 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:18:08.357984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.357956 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:18:08.383540 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.383515 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:18:08.494918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.494888 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.494949 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495000 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495024 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495078 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495102 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495162 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495202 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495252 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495300 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495343 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495383 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495423 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495450 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495533 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495579 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\") pod \"08567cb6-11cc-4575-80c2-001beabfcbae\" (UID: \"08567cb6-11cc-4575-80c2-001beabfcbae\") " Nov 03 19:18:08.495704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495635 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.495704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495666 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\") pod \"b84cac97-525e-49e1-9006-f9b39c1d2439\" (UID: \"b84cac97-525e-49e1-9006-f9b39c1d2439\") " Nov 03 19:18:08.496059 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495419 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496059 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496038 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496059 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495457 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496177 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495611 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495668 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495804 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495925 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.495894 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496277 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496477 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496499 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496516 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496529 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496544 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496557 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.496572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496570 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.497035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496583 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84cac97-525e-49e1-9006-f9b39c1d2439-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.497035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496597 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.497035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.496610 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08567cb6-11cc-4575-80c2-001beabfcbae-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.497704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.497655 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:08.497863 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.497822 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:18:08.497939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.497878 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.498454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.498427 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:08.498691 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.498664 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8" pod "08567cb6-11cc-4575-80c2-001beabfcbae" (UID: "08567cb6-11cc-4575-80c2-001beabfcbae"). InnerVolumeSpecName "kube-api-access-tn6s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:08.498808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.498787 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:08.498848 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.498819 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "kube-api-access-kdkjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:08.499249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.499232 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo" pod "b84cac97-525e-49e1-9006-f9b39c1d2439" (UID: "b84cac97-525e-49e1-9006-f9b39c1d2439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:18:08.598111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598071 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6s8\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-kube-api-access-tn6s8\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598105 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08567cb6-11cc-4575-80c2-001beabfcbae-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598134 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08567cb6-11cc-4575-80c2-001beabfcbae-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598148 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdkjm\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-kube-api-access-kdkjm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598163 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598175 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598187 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84cac97-525e-49e1-9006-f9b39c1d2439-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:08.598418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:08.598198 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08567cb6-11cc-4575-80c2-001beabfcbae-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:09.139448 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.139412 5209 generic.go:356] "Generic (PLEG): container finished" podID="08567cb6-11cc-4575-80c2-001beabfcbae" containerID="5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a" exitCode=0 Nov 03 19:18:09.139649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.139470 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" Nov 03 19:18:09.139649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.139495 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" event={"ID":"08567cb6-11cc-4575-80c2-001beabfcbae","Type":"ContainerDied","Data":"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a"} Nov 03 19:18:09.139649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.139551 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg" event={"ID":"08567cb6-11cc-4575-80c2-001beabfcbae","Type":"ContainerDied","Data":"695598ed758c23a9a865bd12d6a6b0481b7fd8e9dd5111a3bbbb6df1eabd68e2"} Nov 03 19:18:09.139649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.139581 5209 scope.go:122] "RemoveContainer" containerID="5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a" Nov 03 19:18:09.141654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.141069 5209 generic.go:356] "Generic (PLEG): container finished" podID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerID="4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974" exitCode=0 Nov 03 19:18:09.141654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.141110 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" Nov 03 19:18:09.141654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.141157 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" event={"ID":"b84cac97-525e-49e1-9006-f9b39c1d2439","Type":"ContainerDied","Data":"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974"} Nov 03 19:18:09.141654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.141245 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb" event={"ID":"b84cac97-525e-49e1-9006-f9b39c1d2439","Type":"ContainerDied","Data":"d132a41cebf7fb5e795ba64cf3469dba47869642ada7b39a68daf491c775cb10"} Nov 03 19:18:09.150049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.150030 5209 scope.go:122] "RemoveContainer" containerID="5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a" Nov 03 19:18:09.150363 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:18:09.150322 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a\": container with ID starting with 5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a not found: ID does not exist" containerID="5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a" Nov 03 19:18:09.150561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.150361 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a"} err="failed to get container status \"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a\": rpc error: code = NotFound desc = could not find container \"5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a\": container with ID starting with 5431b9a5cf849c953840a2e8c3a9ae278725bbadad135b0464942f9cc8a1071a not found: ID does not exist" Nov 03 19:18:09.150561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.150386 5209 scope.go:122] "RemoveContainer" containerID="4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974" Nov 03 19:18:09.160774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.160747 5209 scope.go:122] "RemoveContainer" containerID="4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974" Nov 03 19:18:09.161181 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:18:09.161105 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974\": container with ID starting with 4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974 not found: ID does not exist" containerID="4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974" Nov 03 19:18:09.161181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.161168 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974"} err="failed to get container status \"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974\": rpc error: code = NotFound desc = could not find container \"4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974\": container with ID starting with 4c7f4e7c249c62ccc298728a85a0ef7be1b5e4c6c4bc5278c19fa9e65a5b6974 not found: ID does not exist" Nov 03 19:18:09.295910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.295864 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:18:09.342499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.342471 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-v77pg"] Nov 03 19:18:09.406170 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.406077 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:18:09.409805 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:09.409777 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6ntwb"] Nov 03 19:18:10.699463 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:10.699433 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" path="/var/lib/kubelet/pods/08567cb6-11cc-4575-80c2-001beabfcbae/volumes" Nov 03 19:18:10.699920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:10.699905 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" path="/var/lib/kubelet/pods/b84cac97-525e-49e1-9006-f9b39c1d2439/volumes" Nov 03 19:18:14.644185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644148 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:18:14.644682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644669 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644683 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644694 5209 cpu_manager.go:414] "containerMap: removing container" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644699 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644706 5209 cpu_manager.go:414] "containerMap: removing container" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644711 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" Nov 03 19:18:14.644721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644720 5209 cpu_manager.go:414] "containerMap: removing container" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644726 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644735 5209 cpu_manager.go:414] "containerMap: removing container" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644740 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644748 5209 cpu_manager.go:414] "containerMap: removing container" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644752 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644761 5209 cpu_manager.go:414] "containerMap: removing container" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644766 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644772 5209 cpu_manager.go:414] "containerMap: removing container" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644776 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644782 5209 cpu_manager.go:414] "containerMap: removing container" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644786 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644797 5209 cpu_manager.go:414] "containerMap: removing container" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644802 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644808 5209 cpu_manager.go:414] "containerMap: removing container" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644812 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644818 5209 cpu_manager.go:414] "containerMap: removing container" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644823 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644829 5209 cpu_manager.go:414] "containerMap: removing container" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644834 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644843 5209 cpu_manager.go:414] "containerMap: removing container" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644847 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644855 5209 cpu_manager.go:414] "containerMap: removing container" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644866 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644880 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644885 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644898 5209 cpu_manager.go:414] "containerMap: removing container" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644903 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644910 5209 cpu_manager.go:414] "containerMap: removing container" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644915 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644923 5209 cpu_manager.go:414] "containerMap: removing container" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644926 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644933 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.644924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.644937 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645035 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645044 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645052 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="376ad674-5c3d-49e7-9697-2dc47b219634" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645058 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="361ba76d-8a0b-4e71-8240-525f86766a44" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645064 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac42e280-2969-42ae-bbc3-68509c342310" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645070 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645076 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="08567cb6-11cc-4575-80c2-001beabfcbae" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645081 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645086 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645091 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645096 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645100 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645105 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a3bba6-2fba-417d-ac01-a1f1448f1eeb" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645111 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a521666-3127-4542-95b0-0d51a945815d" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645130 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6ce26c-0fb2-4bc9-b860-196ef1247bc0" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645138 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84cac97-525e-49e1-9006-f9b39c1d2439" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645145 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dc0ad5-7047-4668-8563-eb5242bb96ec" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645150 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c957be8-bd04-47f2-9089-f309ace6ff11" containerName="istio-proxy" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645156 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645161 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.645864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.645167 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.648307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.648287 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.679142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.679092 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.679303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.679245 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rbb\" (UniqueName: \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.704192 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.703519 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-826a68\"" Nov 03 19:18:14.704192 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.703521 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-826a68-dockercfg-qsx4w\"" Nov 03 19:18:14.707356 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.707328 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:18:14.780080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.780050 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.780254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.780136 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rbb\" (UniqueName: \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.780805 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.780787 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.817811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.817768 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:18:14.818645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818615 5209 cpu_manager.go:414] "containerMap: removing container" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.818645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818641 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c62ca4f-d060-43c8-877c-5128dcc53d73" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818663 5209 cpu_manager.go:414] "containerMap: removing container" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818671 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818683 5209 cpu_manager.go:414] "containerMap: removing container" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818690 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818747 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.818860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818754 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.819200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818959 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df8c1e-cfa1-4de5-814b-2f4acae0308a" containerName="cleanup" Nov 03 19:18:14.819200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818973 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="752b3ff3-82d0-4e3f-aeb3-c9a4ff9bdd6c" containerName="cleanup" Nov 03 19:18:14.819200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.818995 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c365e2-01b4-4104-95ce-61baf6ef5df2" containerName="cleanup" Nov 03 19:18:14.825481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.825445 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:14.844261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.844228 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rbb\" (UniqueName: \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\") pod \"payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.873460 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.873428 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:18:14.882236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.882186 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt8k\" (UniqueName: \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:14.882410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.882262 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:14.967674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.967599 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:14.983531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.983494 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt8k\" (UniqueName: \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:14.984052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.983572 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:14.984563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:14.984536 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:15.047668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:15.047618 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt8k\" (UniqueName: \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\") pod \"payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:15.169174 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:15.169141 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:15.375362 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:15.375331 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:18:15.380432 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:18:15.380392 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d5439_a51f_41c6_99c5_5f914803205f.slice/crio-fdf69e07949a93a1af0032a78200555da352b50a7eb3685b6695ef82b2438a50 WatchSource:0}: Error finding container fdf69e07949a93a1af0032a78200555da352b50a7eb3685b6695ef82b2438a50: Status 404 returned error can't find the container with id fdf69e07949a93a1af0032a78200555da352b50a7eb3685b6695ef82b2438a50 Nov 03 19:18:15.546957 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:15.546868 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:18:15.551576 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:18:15.551546 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0a983e_24ee_4862_bd44_796db6c6079b.slice/crio-b14e2fb88d40fbe49540da30b9a44ab4ab23edc33e14f95a4643b7a12aa635dd WatchSource:0}: Error finding container b14e2fb88d40fbe49540da30b9a44ab4ab23edc33e14f95a4643b7a12aa635dd: Status 404 returned error can't find the container with id b14e2fb88d40fbe49540da30b9a44ab4ab23edc33e14f95a4643b7a12aa635dd Nov 03 19:18:16.045076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.045039 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:18:16.049110 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.049086 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.099296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.098530 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1410ed-openshift-default-dockercfg-df98m\"" Nov 03 19:18:16.101493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.101451 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.101771 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.101753 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.102001 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.101984 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnbb\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.102225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.102208 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.102435 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.102418 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.102643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.102624 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.102833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.102816 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.103023 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.103007 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.103280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.103263 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.160959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.160914 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:18:16.248989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.248938 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.266476 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.266440 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnbb\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.266749 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.266732 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.266896 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.266883 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.267029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.267017 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.267112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.267101 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.267286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.267263 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.267496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.267484 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.267633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.267619 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.271707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.270921 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.274755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.274729 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.276527 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.276475 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.276645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.276544 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.276990 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.276968 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.279000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.278969 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" event={"ID":"4f0a983e-24ee-4862-bd44-796db6c6079b","Type":"ContainerStarted","Data":"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c"} Nov 03 19:18:16.279201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.279179 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" event={"ID":"4f0a983e-24ee-4862-bd44-796db6c6079b","Type":"ContainerStarted","Data":"b14e2fb88d40fbe49540da30b9a44ab4ab23edc33e14f95a4643b7a12aa635dd"} Nov 03 19:18:16.280198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.280104 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:16.281066 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.281017 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.284555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.283758 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" event={"ID":"426d5439-a51f-41c6-99c5-5f914803205f","Type":"ContainerStarted","Data":"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17"} Nov 03 19:18:16.284555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.283803 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" event={"ID":"426d5439-a51f-41c6-99c5-5f914803205f","Type":"ContainerStarted","Data":"fdf69e07949a93a1af0032a78200555da352b50a7eb3685b6695ef82b2438a50"} Nov 03 19:18:16.284555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.284504 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:16.285159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.285140 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.327515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.327485 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.327713 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.327692 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnbb\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-c5s98\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.364854 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.364818 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:16.391517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.391460 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" podStartSLOduration=2.3914427959999998 podStartE2EDuration="2.391442796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:16.389541267 +0000 UTC m=+1977.800641401" watchObservedRunningTime="2026-11-03 19:18:16.391442796 +0000 UTC m=+1977.802542920" Nov 03 19:18:16.518929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.518873 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" podStartSLOduration=2.51885635 podStartE2EDuration="2.51885635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:16.511679517 +0000 UTC m=+1977.922779642" watchObservedRunningTime="2026-11-03 19:18:16.51885635 +0000 UTC m=+1977.929956473" Nov 03 19:18:16.837317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.837283 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:18:16.872365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.867908 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:16.872365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.867998 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:16.872365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:16.868038 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:17.197541 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:17.197450 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:18:17.300993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:17.300954 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerStarted","Data":"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1"} Nov 03 19:18:17.301266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:17.301238 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerStarted","Data":"dfb1827d2a326ec0fc5d5a18ec9ef284f27de878122d05b564b84cef5501a506"} Nov 03 19:18:17.365592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:17.365557 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:17.418055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:17.418005 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podStartSLOduration=1.417990417 podStartE2EDuration="1.417990417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:17.415300884 +0000 UTC m=+1978.826401052" watchObservedRunningTime="2026-11-03 19:18:17.417990417 +0000 UTC m=+1978.829090540" Nov 03 19:18:18.382177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:18.381402 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:19.322687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:19.322655 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:19.323455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:19.323438 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:18:27.304239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:27.304213 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:18:28.320337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:28.320308 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:18:29.457557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.457482 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:18:29.463252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.463231 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.522466 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.522422 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:18:29.562897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.562863 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.563080 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.563059 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r6k\" (UniqueName: \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.564088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.564068 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1410ed-dockercfg-6hfp4\"" Nov 03 19:18:29.603949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.603924 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1410ed\"" Nov 03 19:18:29.664829 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.664799 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.664992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.664914 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r6k\" (UniqueName: \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.665558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.665535 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.723275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.723207 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r6k\" (UniqueName: \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\") pod \"payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:29.781533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:29.781505 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:30.154599 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:30.154540 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:18:30.159073 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:18:30.159049 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8f51c2_3bfb_44ee_9e3e_75491321966b.slice/crio-9d7bb82111ad884b7aa21b7fcc6983df1b1587bc6ee010bf2bf8cb8c9971d7cb WatchSource:0}: Error finding container 9d7bb82111ad884b7aa21b7fcc6983df1b1587bc6ee010bf2bf8cb8c9971d7cb: Status 404 returned error can't find the container with id 9d7bb82111ad884b7aa21b7fcc6983df1b1587bc6ee010bf2bf8cb8c9971d7cb Nov 03 19:18:30.399610 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:30.399567 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" event={"ID":"3b8f51c2-3bfb-44ee-9e3e-75491321966b","Type":"ContainerStarted","Data":"0d60cfbbaf72d10d2c0aa720d9b655c7bf137d7993f2e9e9c0d915322464cf64"} Nov 03 19:18:30.399765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:30.399633 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:18:30.399765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:30.399650 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" event={"ID":"3b8f51c2-3bfb-44ee-9e3e-75491321966b","Type":"ContainerStarted","Data":"9d7bb82111ad884b7aa21b7fcc6983df1b1587bc6ee010bf2bf8cb8c9971d7cb"} Nov 03 19:18:30.510164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:30.510048 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" podStartSLOduration=1.510031556 podStartE2EDuration="1.510031556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:30.507960299 +0000 UTC m=+1991.919060421" watchObservedRunningTime="2026-11-03 19:18:30.510031556 +0000 UTC m=+1991.921131710" Nov 03 19:18:31.937357 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:18:31.937320 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:18:41.409109 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:18:41.409075 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:19:00.561512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.561474 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:19:00.561937 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.561712 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerName="discovery" containerID="cri-o://f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876" gracePeriod=30 Nov 03 19:19:00.846095 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.846068 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:19:00.921015 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.920981 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921055 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921095 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921172 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921227 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921248 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921441 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\") pod \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\" (UID: \"262e4b92-2faa-4dea-8a7a-0804d0aba4d1\") " Nov 03 19:19:00.921509 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.921467 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:00.922235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.922204 5209 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-ca-configmap\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:00.923626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923596 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:00.923732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923659 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:00.923732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923712 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:00.923840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923801 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:00.923907 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923890 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:00.923945 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:00.923904 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf" pod "262e4b92-2faa-4dea-8a7a-0804d0aba4d1" (UID: "262e4b92-2faa-4dea-8a7a-0804d0aba4d1"). InnerVolumeSpecName "kube-api-access-9jfcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:01.023842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023809 5209 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-csr-dns-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.023842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023837 5209 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-local-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.023842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023846 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jfcf\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-kube-api-access-9jfcf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.024064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023855 5209 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-cacerts\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.024064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023864 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.024064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.023872 5209 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/262e4b92-2faa-4dea-8a7a-0804d0aba4d1-istio-kubeconfig\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:01.645747 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.645713 5209 generic.go:356] "Generic (PLEG): container finished" podID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerID="f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876" exitCode=0 Nov 03 19:19:01.646231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.645758 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" Nov 03 19:19:01.646231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.645791 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" event={"ID":"262e4b92-2faa-4dea-8a7a-0804d0aba4d1","Type":"ContainerDied","Data":"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876"} Nov 03 19:19:01.646231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.645845 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx" event={"ID":"262e4b92-2faa-4dea-8a7a-0804d0aba4d1","Type":"ContainerDied","Data":"a373c3812b10d201857c44fd0dc0ff6f88e7e4b20cbee7a7a3c2c8c300bf0fc7"} Nov 03 19:19:01.646231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.645864 5209 scope.go:122] "RemoveContainer" containerID="f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876" Nov 03 19:19:01.656381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.656365 5209 scope.go:122] "RemoveContainer" containerID="f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876" Nov 03 19:19:01.656660 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:01.656642 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876\": container with ID starting with f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876 not found: ID does not exist" containerID="f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876" Nov 03 19:19:01.656715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.656668 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876"} err="failed to get container status \"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876\": rpc error: code = NotFound desc = could not find container \"f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876\": container with ID starting with f6d3456add83d221748447bfcbebb468762696fd7dfb4f7b438c548d44329876 not found: ID does not exist" Nov 03 19:19:01.850551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.850517 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:19:01.854225 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:01.854201 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8glmx"] Nov 03 19:19:01.938298 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:01.938220 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:19:02.700751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:02.700713 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" path="/var/lib/kubelet/pods/262e4b92-2faa-4dea-8a7a-0804d0aba4d1/volumes" Nov 03 19:19:09.517025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:09.516986 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:19:31.943187 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:31.942919 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:19:33.760673 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:33.760635 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:19:33.761147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:33.760843 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" containerID="cri-o://2892bbf248f4101f07febe3709873c546010f05c7c55e883f392f613ee47f42d" gracePeriod=30 Nov 03 19:19:35.723161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:35.723099 5209 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:19:35.723538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:35.723183 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:19:36.681005 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.680974 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:19:36.681572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.681557 5209 cpu_manager.go:414] "containerMap: removing container" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerName="discovery" Nov 03 19:19:36.681645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.681574 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerName="discovery" Nov 03 19:19:36.681712 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.681700 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="262e4b92-2faa-4dea-8a7a-0804d0aba4d1" containerName="discovery" Nov 03 19:19:36.686329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.686309 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.715185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.715155 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.715330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.715261 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.715434 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.715418 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhlx\" (UniqueName: \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.715477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.715468 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.737317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.737292 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-826a68-serving-cert\"" Nov 03 19:19:36.789830 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.788422 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:19:36.817341 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.817304 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhlx\" (UniqueName: \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.817525 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.817404 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.817525 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.817474 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.817652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.817538 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.817726 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:36.817703 5209 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-serving-cert: secret "maas-api-e2e-shared-b-w3-826a68-serving-cert" not found Nov 03 19:19:36.817789 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:36.817777 5209 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls podName:1cde17f5-2789-4f0c-a8b4-6377a72ab877 nodeName:}" failed. No retries permitted until 2026-11-03 19:19:37.317756341 +0000 UTC m=+2058.728856443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls") pod "maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" (UID: "1cde17f5-2789-4f0c-a8b4-6377a72ab877") : secret "maas-api-e2e-shared-b-w3-826a68-serving-cert" not found Nov 03 19:19:36.822809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.822768 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.824701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.824672 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:36.876201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:36.876174 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhlx\" (UniqueName: \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:37.333202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:37.327964 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:37.343994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:37.343962 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:37.600268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:37.600113 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:38.007049 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:38.006971 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:19:38.011582 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:19:38.011555 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cde17f5_2789_4f0c_a8b4_6377a72ab877.slice/crio-1c68029d93a7408818adbdeac8cba834fe9cb97873c4dd374b290c7c823f3dd5 WatchSource:0}: Error finding container 1c68029d93a7408818adbdeac8cba834fe9cb97873c4dd374b290c7c823f3dd5: Status 404 returned error can't find the container with id 1c68029d93a7408818adbdeac8cba834fe9cb97873c4dd374b290c7c823f3dd5 Nov 03 19:19:38.013374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:38.013356 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:19:38.889904 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:38.889872 5209 generic.go:356] "Generic (PLEG): container finished" podID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerID="2892bbf248f4101f07febe3709873c546010f05c7c55e883f392f613ee47f42d" exitCode=0 Nov 03 19:19:38.890045 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:38.889980 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" event={"ID":"39dd13a0-d8e0-4b18-886a-e914300015ac","Type":"ContainerDied","Data":"2892bbf248f4101f07febe3709873c546010f05c7c55e883f392f613ee47f42d"} Nov 03 19:19:38.891579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:38.891546 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" event={"ID":"1cde17f5-2789-4f0c-a8b4-6377a72ab877","Type":"ContainerStarted","Data":"1c68029d93a7408818adbdeac8cba834fe9cb97873c4dd374b290c7c823f3dd5"} Nov 03 19:19:39.075683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.075658 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:19:39.160739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160707 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.160931 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160774 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.160931 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160802 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.160931 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160821 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.160931 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160848 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.161166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.160937 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.161166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161023 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.161166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161073 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.161166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161164 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\") pod \"39dd13a0-d8e0-4b18-886a-e914300015ac\" (UID: \"39dd13a0-d8e0-4b18-886a-e914300015ac\") " Nov 03 19:19:39.161677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161158 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.161677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161361 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.161677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161497 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.161896 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161701 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.161896 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161862 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:39.162011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161963 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd13a0-d8e0-4b18-886a-e914300015ac-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.162011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161980 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.162011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.161997 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.162227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.162010 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.162227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.162025 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.163303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.163253 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:39.163475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.163456 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "kube-api-access-zvskz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:39.163595 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.163571 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.164275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.164254 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo" pod "39dd13a0-d8e0-4b18-886a-e914300015ac" (UID: "39dd13a0-d8e0-4b18-886a-e914300015ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:39.263337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.263295 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvskz\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-kube-api-access-zvskz\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.263337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.263326 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.263337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.263337 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.263337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.263345 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd13a0-d8e0-4b18-886a-e914300015ac-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.900243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.900211 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" Nov 03 19:19:39.900243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.900229 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx" event={"ID":"39dd13a0-d8e0-4b18-886a-e914300015ac","Type":"ContainerDied","Data":"2923b120b5cd52a0b58325df1ab115973a6b9eda87aa462709c2b42a14478e67"} Nov 03 19:19:39.900484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.900294 5209 scope.go:122] "RemoveContainer" containerID="2892bbf248f4101f07febe3709873c546010f05c7c55e883f392f613ee47f42d" Nov 03 19:19:39.901811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.901789 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" event={"ID":"1cde17f5-2789-4f0c-a8b4-6377a72ab877","Type":"ContainerStarted","Data":"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8"} Nov 03 19:19:39.901954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:39.901842 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:40.103438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:40.103407 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:19:40.107812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:40.107785 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-nmlcx"] Nov 03 19:19:40.218564 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:40.218458 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" podStartSLOduration=3.257829534 podStartE2EDuration="4.218444232s" totalImagesPullingTime="960.614698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:40.215464258 +0000 UTC m=+2061.626564382" watchObservedRunningTime="2026-11-03 19:19:40.218444232 +0000 UTC m=+2061.629544384" Nov 03 19:19:40.699605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:40.699573 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" path="/var/lib/kubelet/pods/39dd13a0-d8e0-4b18-886a-e914300015ac/volumes" Nov 03 19:19:45.916112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:45.916084 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:19:45.972587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:45.972556 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:19:45.972831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:45.972794 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" containerID="cri-o://953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1" gracePeriod=30 Nov 03 19:19:48.831313 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:48.831280 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:19:48.831819 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:48.831563 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" containerID="cri-o://bf7005bf7a863799407dc8f043156c232f66bdbda0b1ee1958a8a09dab054ea1" gracePeriod=30 Nov 03 19:19:50.297565 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:50.297527 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:19:50.298028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:50.297592 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:19:51.271769 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.271742 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:19:51.428515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428480 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428550 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428593 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428629 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428684 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428774 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428908 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.428966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428944 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:51.429459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.428967 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:51.429459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429016 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:51.429459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429043 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.429459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429091 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\") pod \"4797a113-bf51-4ed9-a506-a00be7473743\" (UID: \"4797a113-bf51-4ed9-a506-a00be7473743\") " Nov 03 19:19:51.429459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429270 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:51.429726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429469 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:51.429726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429706 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.429836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429728 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4797a113-bf51-4ed9-a506-a00be7473743-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.429836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429744 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.429836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429758 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.429836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.429773 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.431239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.431203 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:51.431963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.431769 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:51.432133 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.432092 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "kube-api-access-pv2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:51.432212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.432183 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token" pod "4797a113-bf51-4ed9-a506-a00be7473743" (UID: "4797a113-bf51-4ed9-a506-a00be7473743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:51.530760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.530687 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.530760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.530713 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4797a113-bf51-4ed9-a506-a00be7473743-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.530760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.530723 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2xr\" (UniqueName: \"kubernetes.io/projected/4797a113-bf51-4ed9-a506-a00be7473743-kube-api-access-pv2xr\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.530760 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.530734 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4797a113-bf51-4ed9-a506-a00be7473743-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:51.860151 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.860109 5209 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:19:51.860300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.860184 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:19:51.993022 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.992986 5209 generic.go:356] "Generic (PLEG): container finished" podID="4797a113-bf51-4ed9-a506-a00be7473743" containerID="953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1" exitCode=0 Nov 03 19:19:51.993245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.993049 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" Nov 03 19:19:51.993245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.993110 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" event={"ID":"4797a113-bf51-4ed9-a506-a00be7473743","Type":"ContainerDied","Data":"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1"} Nov 03 19:19:51.993245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.993200 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz" event={"ID":"4797a113-bf51-4ed9-a506-a00be7473743","Type":"ContainerDied","Data":"bb688e833f4b88079bcc67ef80f46d40b45dd3d9669b47088b18e274227eb1c7"} Nov 03 19:19:51.993245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:51.993229 5209 scope.go:122] "RemoveContainer" containerID="953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1" Nov 03 19:19:52.004966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:52.004950 5209 scope.go:122] "RemoveContainer" containerID="953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1" Nov 03 19:19:52.005249 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:19:52.005229 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1\": container with ID starting with 953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1 not found: ID does not exist" containerID="953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1" Nov 03 19:19:52.005315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:52.005255 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1"} err="failed to get container status \"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1\": rpc error: code = NotFound desc = could not find container \"953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1\": container with ID starting with 953c00a98c93aac8e89e5b10911ec66b08dfdb4635548605ad75ae9c48b075e1 not found: ID does not exist" Nov 03 19:19:52.179047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:52.178950 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:19:52.196558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:52.196529 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-rfptz"] Nov 03 19:19:52.702441 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:52.702408 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4797a113-bf51-4ed9-a506-a00be7473743" path="/var/lib/kubelet/pods/4797a113-bf51-4ed9-a506-a00be7473743/volumes" Nov 03 19:19:54.036372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.036338 5209 generic.go:356] "Generic (PLEG): container finished" podID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerID="bf7005bf7a863799407dc8f043156c232f66bdbda0b1ee1958a8a09dab054ea1" exitCode=0 Nov 03 19:19:54.037032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.037005 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" event={"ID":"25f6ba93-1fb3-4743-ae83-0c856a53d2b8","Type":"ContainerDied","Data":"bf7005bf7a863799407dc8f043156c232f66bdbda0b1ee1958a8a09dab054ea1"} Nov 03 19:19:54.195656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.195626 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:19:54.369928 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.369892 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370002 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370064 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370112 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370179 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370356 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370285 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370360 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370418 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.370465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.370455 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\") pod \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\" (UID: \"25f6ba93-1fb3-4743-ae83-0c856a53d2b8\") " Nov 03 19:19:54.372607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.371655 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:54.372607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.371924 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:54.372607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.372104 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:54.372607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.372295 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:54.373412 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.373359 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:54.384426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.384300 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:54.384426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.384322 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:54.384426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.384336 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:54.384426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.384363 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k" pod "25f6ba93-1fb3-4743-ae83-0c856a53d2b8" (UID: "25f6ba93-1fb3-4743-ae83-0c856a53d2b8"). InnerVolumeSpecName "kube-api-access-tpn5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472778 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472812 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472824 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472837 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472858 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472870 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpn5k\" (UniqueName: \"kubernetes.io/projected/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-kube-api-access-tpn5k\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.472880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472886 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.473314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472895 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:54.473314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:54.472905 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f6ba93-1fb3-4743-ae83-0c856a53d2b8-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.046127 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:55.046096 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" Nov 03 19:19:55.046623 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:55.046135 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs" event={"ID":"25f6ba93-1fb3-4743-ae83-0c856a53d2b8","Type":"ContainerDied","Data":"407b35ad832ab5ec4ac997479e8afdbd591af515fa695c3d09ad556d7111c43c"} Nov 03 19:19:55.046623 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:55.046195 5209 scope.go:122] "RemoveContainer" containerID="bf7005bf7a863799407dc8f043156c232f66bdbda0b1ee1958a8a09dab054ea1" Nov 03 19:19:55.248862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:55.248818 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:19:55.294859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:55.294822 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jbmcs"] Nov 03 19:19:56.699671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:56.699637 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" path="/var/lib/kubelet/pods/25f6ba93-1fb3-4743-ae83-0c856a53d2b8/volumes" Nov 03 19:19:57.481029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.480992 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:19:57.481620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481601 5209 cpu_manager.go:414] "containerMap: removing container" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" Nov 03 19:19:57.481620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481620 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" Nov 03 19:19:57.481756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481660 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" Nov 03 19:19:57.481756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481667 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" Nov 03 19:19:57.481756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481681 5209 cpu_manager.go:414] "containerMap: removing container" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" Nov 03 19:19:57.481756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481686 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" Nov 03 19:19:57.481966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481760 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f6ba93-1fb3-4743-ae83-0c856a53d2b8" containerName="istio-proxy" Nov 03 19:19:57.481966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481773 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4797a113-bf51-4ed9-a506-a00be7473743" containerName="istio-proxy" Nov 03 19:19:57.481966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.481780 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dd13a0-d8e0-4b18-886a-e914300015ac" containerName="istio-proxy" Nov 03 19:19:57.486520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.486498 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.557367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.557335 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d2cbe3-openshift-default-dockercfg-x45k8\"" Nov 03 19:19:57.629177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.615923 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629268 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629335 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629401 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629452 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629496 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629547 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629699 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.629775 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.629761 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpn9\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.669979 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.669940 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:19:57.730602 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730570 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730640 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730687 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730725 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730770 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730893 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.730942 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpn9\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731008 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731049 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731102 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731086 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731295 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731007 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731557 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.731626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.731588 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.733849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.733825 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.733983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.733963 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.799682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.799650 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:57.800198 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:57.800171 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpn9\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\") pod \"e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:58.100158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:58.100113 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:58.555029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:58.555001 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:19:58.557411 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:19:58.557384 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f712aea_e9d3_4924_88d7_11bff330aabc.slice/crio-7c9ef713467835d28d0ee9975a378e007b56ed9e049598f7d5f7a22ad21357d9 WatchSource:0}: Error finding container 7c9ef713467835d28d0ee9975a378e007b56ed9e049598f7d5f7a22ad21357d9: Status 404 returned error can't find the container with id 7c9ef713467835d28d0ee9975a378e007b56ed9e049598f7d5f7a22ad21357d9 Nov 03 19:19:58.560168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:58.560133 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:19:58.560255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:58.560216 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:19:58.560294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:58.560263 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:19:59.077285 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:59.077183 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerStarted","Data":"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8"} Nov 03 19:19:59.077285 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:59.077248 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerStarted","Data":"7c9ef713467835d28d0ee9975a378e007b56ed9e049598f7d5f7a22ad21357d9"} Nov 03 19:19:59.100960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:59.100935 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:59.106566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:59.106544 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:19:59.194263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:19:59.194215 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podStartSLOduration=2.19420171 podStartE2EDuration="2.19420171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:59.190086029 +0000 UTC m=+2080.601186152" watchObservedRunningTime="2026-11-03 19:19:59.19420171 +0000 UTC m=+2080.605301843" Nov 03 19:20:00.083850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:00.083819 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:20:00.084831 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:00.084810 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:20:01.937994 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:01.937959 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:20:05.960144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:05.959301 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:20:05.964493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:05.964464 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.025808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.025768 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:20:06.043202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.043164 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5927\" (UniqueName: \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.043390 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.043264 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.053648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.053617 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d2cbe3\"" Nov 03 19:20:06.078311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.078280 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:20:06.083275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.083253 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.100229 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.100206 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d2cbe3-dockercfg-hg7ct\"" Nov 03 19:20:06.138613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.138560 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:20:06.161799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.159165 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.161799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.159331 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.161799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.159472 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.161799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.159518 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5927\" (UniqueName: \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.163378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.163349 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.217327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.217264 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5927\" (UniqueName: \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\") pod \"payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.261336 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.261300 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.261522 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.261453 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.262111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.262092 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.312114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.312078 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:06.319744 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.319719 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\") pod \"payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.397216 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.397187 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:06.686784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.686752 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:20:06.690843 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:20:06.690815 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7357c41d_2486_4dee_90f8_a767aae4f52b.slice/crio-70539b2004228dd21a886f4f8fc926f6c071b8433a11255aaed8ad9918044e3e WatchSource:0}: Error finding container 70539b2004228dd21a886f4f8fc926f6c071b8433a11255aaed8ad9918044e3e: Status 404 returned error can't find the container with id 70539b2004228dd21a886f4f8fc926f6c071b8433a11255aaed8ad9918044e3e Nov 03 19:20:06.798164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:06.798135 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:20:06.802560 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:20:06.802535 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2a92d9_5f8d_4f99_be10_cd54a4ee2f0f.slice/crio-4d98c62669bc0c42b2cb6455f942214bca55122f1e7b5b138dbf7aa31366e3fe WatchSource:0}: Error finding container 4d98c62669bc0c42b2cb6455f942214bca55122f1e7b5b138dbf7aa31366e3fe: Status 404 returned error can't find the container with id 4d98c62669bc0c42b2cb6455f942214bca55122f1e7b5b138dbf7aa31366e3fe Nov 03 19:20:07.128740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.128707 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" event={"ID":"7357c41d-2486-4dee-90f8-a767aae4f52b","Type":"ContainerStarted","Data":"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09"} Nov 03 19:20:07.129256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.128758 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" event={"ID":"7357c41d-2486-4dee-90f8-a767aae4f52b","Type":"ContainerStarted","Data":"70539b2004228dd21a886f4f8fc926f6c071b8433a11255aaed8ad9918044e3e"} Nov 03 19:20:07.129256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.128913 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:07.130504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.130470 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" event={"ID":"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f","Type":"ContainerStarted","Data":"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b"} Nov 03 19:20:07.130620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.130511 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" event={"ID":"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f","Type":"ContainerStarted","Data":"4d98c62669bc0c42b2cb6455f942214bca55122f1e7b5b138dbf7aa31366e3fe"} Nov 03 19:20:07.130620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.130541 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:07.238874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.238819 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" podStartSLOduration=2.23880287 podStartE2EDuration="2.23880287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:07.236550206 +0000 UTC m=+2088.647650328" watchObservedRunningTime="2026-11-03 19:20:07.23880287 +0000 UTC m=+2088.649902996" Nov 03 19:20:07.347661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:07.347609 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" podStartSLOduration=1.347596439 podStartE2EDuration="1.347596439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:07.344652575 +0000 UTC m=+2088.755752699" watchObservedRunningTime="2026-11-03 19:20:07.347596439 +0000 UTC m=+2088.758696562" Nov 03 19:20:18.139613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:18.139579 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:20:18.140035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:18.139664 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:20:18.475766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:18.475681 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:20:21.208683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.208632 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:20:21.213839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.213819 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.263184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.263148 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:20:21.332679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.332605 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d86d1cd0-dockercfg-snsvn\"" Nov 03 19:20:21.346916 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.346871 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d86d1cd0\"" Nov 03 19:20:21.425803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.425760 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.430177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.427502 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88x2\" (UniqueName: \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.436206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.436179 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:20:21.440395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.440374 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.443077 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.443053 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:20:21.529376 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.529275 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88x2\" (UniqueName: \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.529547 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.529381 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwxz\" (UniqueName: \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.529604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.529546 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.529638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.529605 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.530342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.530325 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.592145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.590813 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88x2\" (UniqueName: \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:21.630681 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.630641 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwxz\" (UniqueName: \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.630842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.630763 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.631534 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.631515 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.697139 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.696792 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwxz\" (UniqueName: \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.759164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.759139 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:21.837928 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:21.837766 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:22.163683 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:22.163609 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:20:22.167973 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:20:22.167948 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd82e045_4677_4ffb_8e62_52d4b6973595.slice/crio-c1f252cec6d8622b932e1a00f361933c7790b5a495db399aa3f603b7dcad1ad7 WatchSource:0}: Error finding container c1f252cec6d8622b932e1a00f361933c7790b5a495db399aa3f603b7dcad1ad7: Status 404 returned error can't find the container with id c1f252cec6d8622b932e1a00f361933c7790b5a495db399aa3f603b7dcad1ad7 Nov 03 19:20:22.240003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:22.239977 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" event={"ID":"fd82e045-4677-4ffb-8e62-52d4b6973595","Type":"ContainerStarted","Data":"c1f252cec6d8622b932e1a00f361933c7790b5a495db399aa3f603b7dcad1ad7"} Nov 03 19:20:22.275410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:22.275385 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:20:22.304713 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:20:22.304660 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42ae4f9_4c74_46a0_a076_65980290888c.slice/crio-266a086418817dff96794010c0b067a264862a5ccf7ed6d668f5442dc777bf3e WatchSource:0}: Error finding container 266a086418817dff96794010c0b067a264862a5ccf7ed6d668f5442dc777bf3e: Status 404 returned error can't find the container with id 266a086418817dff96794010c0b067a264862a5ccf7ed6d668f5442dc777bf3e Nov 03 19:20:23.248147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.248097 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" event={"ID":"c42ae4f9-4c74-46a0-a076-65980290888c","Type":"ContainerStarted","Data":"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc"} Nov 03 19:20:23.248687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.248178 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" event={"ID":"c42ae4f9-4c74-46a0-a076-65980290888c","Type":"ContainerStarted","Data":"266a086418817dff96794010c0b067a264862a5ccf7ed6d668f5442dc777bf3e"} Nov 03 19:20:23.248687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.248208 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:23.250083 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.250055 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" event={"ID":"fd82e045-4677-4ffb-8e62-52d4b6973595","Type":"ContainerStarted","Data":"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c"} Nov 03 19:20:23.250205 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.250151 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:23.359666 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.359610 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" podStartSLOduration=2.359595348 podStartE2EDuration="2.359595348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:23.3572781 +0000 UTC m=+2104.768378242" watchObservedRunningTime="2026-11-03 19:20:23.359595348 +0000 UTC m=+2104.770695471" Nov 03 19:20:23.469713 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:23.469655 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" podStartSLOduration=2.469640566 podStartE2EDuration="2.469640566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:23.466805325 +0000 UTC m=+2104.877905469" watchObservedRunningTime="2026-11-03 19:20:23.469640566 +0000 UTC m=+2104.880740711" Nov 03 19:20:30.300287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.300174 5209 generic.go:356] "Generic (PLEG): container finished" podID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerID="287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1" exitCode=0 Nov 03 19:20:30.300287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.300247 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerDied","Data":"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1"} Nov 03 19:20:30.300902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.300713 5209 scope.go:122] "RemoveContainer" containerID="287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1" Nov 03 19:20:30.302442 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.302416 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:20:30.302546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.302481 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:20:30.302546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:30.302520 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:20:31.309346 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:31.309309 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerStarted","Data":"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e"} Nov 03 19:20:31.365837 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:31.365808 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:20:31.367054 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:31.367028 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:31.367178 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:31.367082 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:31.937690 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:31.937646 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:20:32.365925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:32.365891 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:32.366337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:32.365942 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:33.365913 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:33.365877 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:33.366088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:33.365929 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:34.259144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:34.259097 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:20:34.259328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:34.259189 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:20:34.322784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:34.322755 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:20:34.365742 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:34.365695 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:34.365879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:34.365764 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:35.365867 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:35.365829 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:35.366296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:35.365884 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:36.365454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:36.365420 5209 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:20:36.365661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:36.365478 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 19:20:37.369838 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:37.369810 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:20:37.371081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:37.371056 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:20:48.928948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:48.928910 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:20:48.929469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:48.929286 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" containerID="cri-o://b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7" gracePeriod=30 Nov 03 19:20:48.975534 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:48.975508 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:20:49.409430 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:49.409387 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:20:49.409704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:49.409669 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" containerID="cri-o://e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b" gracePeriod=30 Nov 03 19:20:49.440630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:49.440592 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" containerID="cri-o://c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb" gracePeriod=30 Nov 03 19:20:49.738235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:49.738164 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:20:49.738235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:49.738217 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:20:50.643668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:50.643632 5209 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:20:50.644067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:50.643686 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:20:50.963350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:50.963270 5209 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:20:50.963350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:50.963325 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:20:51.118643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.118606 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:20:51.118839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.118806 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" containerID="cri-o://f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312" gracePeriod=30 Nov 03 19:20:51.166062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.166033 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 19:20:51.166312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.166289 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" containerID="cri-o://8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85" gracePeriod=30 Nov 03 19:20:51.172511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.172469 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:20:51.172806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.172758 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" containerID="cri-o://d9b84ffa76726360feb3b558b835e7170c22060abc13023a7a75a977f0684ce8" gracePeriod=30 Nov 03 19:20:51.212204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.212175 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:20:51.212402 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:51.212380 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" containerID="cri-o://0bab422f2f553187efc1d45145248f159175a944d32e109044f4dcdf84342ea0" gracePeriod=30 Nov 03 19:20:54.207407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.207332 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:20:54.306084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306057 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306096 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306171 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306258 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306282 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306358 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306363 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.306530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306393 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306430 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306493 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\") pod \"b42d275d-beb4-4789-861b-83632fbba289\" (UID: \"b42d275d-beb4-4789-861b-83632fbba289\") " Nov 03 19:20:54.306788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306563 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.306788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306603 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.306897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306827 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.306897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.306865 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:54.307162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.307138 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.307295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.307164 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.307295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.307180 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d275d-beb4-4789-861b-83632fbba289-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.307295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.307193 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.307295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.307206 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.308709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.308684 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.308709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.308696 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:54.308828 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.308735 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.308828 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.308773 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279" pod "b42d275d-beb4-4789-861b-83632fbba289" (UID: "b42d275d-beb4-4789-861b-83632fbba289"). InnerVolumeSpecName "kube-api-access-sh279". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.408977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.408939 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh279\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-kube-api-access-sh279\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.408977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.408967 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d275d-beb4-4789-861b-83632fbba289-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.408977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.408977 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d275d-beb4-4789-861b-83632fbba289-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.408977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.408987 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d275d-beb4-4789-861b-83632fbba289-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.474648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.474620 5209 generic.go:356] "Generic (PLEG): container finished" podID="b42d275d-beb4-4789-861b-83632fbba289" containerID="b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7" exitCode=0 Nov 03 19:20:54.474768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.474666 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" Nov 03 19:20:54.474768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.474699 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" event={"ID":"b42d275d-beb4-4789-861b-83632fbba289","Type":"ContainerDied","Data":"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7"} Nov 03 19:20:54.474768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.474761 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl" event={"ID":"b42d275d-beb4-4789-861b-83632fbba289","Type":"ContainerDied","Data":"33828bccdb18520d420579865bc63df00bc8fa2274599f984ef159bbbaae0e0f"} Nov 03 19:20:54.474940 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.474793 5209 scope.go:122] "RemoveContainer" containerID="b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7" Nov 03 19:20:54.553805 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.553775 5209 scope.go:122] "RemoveContainer" containerID="b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7" Nov 03 19:20:54.554243 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:54.554217 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7\": container with ID starting with b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7 not found: ID does not exist" containerID="b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7" Nov 03 19:20:54.554344 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.554251 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7"} err="failed to get container status \"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7\": rpc error: code = NotFound desc = could not find container \"b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7\": container with ID starting with b0da4f43a59dcaee823371f2df2fc7e7efdc5a79cefac17f417eda1f590215a7 not found: ID does not exist" Nov 03 19:20:54.633791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.633755 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:20:54.637163 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.637134 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-5g6zl"] Nov 03 19:20:54.687667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.687644 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:20:54.703416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.703387 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42d275d-beb4-4789-861b-83632fbba289" path="/var/lib/kubelet/pods/b42d275d-beb4-4789-861b-83632fbba289/volumes" Nov 03 19:20:54.745950 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.745929 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:20:54.814977 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.814948 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.814997 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815025 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815172 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815219 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815292 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815390 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.815408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815403 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:54.815614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815417 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815581 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815629 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\") pod \"38aa753f-f902-46e6-9aa6-d64ba51f3472\" (UID: \"38aa753f-f902-46e6-9aa6-d64ba51f3472\") " Nov 03 19:20:54.815719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815636 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.815887 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.815845 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816029 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816735 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816826 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816860 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816877 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aa753f-f902-46e6-9aa6-d64ba51f3472-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.816944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.816891 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.817969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.817943 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.818076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.817981 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:54.818076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.818012 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.819178 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.819150 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh" pod "38aa753f-f902-46e6-9aa6-d64ba51f3472" (UID: "38aa753f-f902-46e6-9aa6-d64ba51f3472"). InnerVolumeSpecName "kube-api-access-jm9qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.918806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.918774 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919163 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919113 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919295 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919216 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919322 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919349 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919407 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919470 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919509 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919565 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\") pod \"b3906bfb-afdb-4752-907f-695607248522\" (UID: \"b3906bfb-afdb-4752-907f-695607248522\") " Nov 03 19:20:54.919633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919597 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.919633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919614 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:54.919821 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.919801 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.920328 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920223 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.920464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920265 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm9qh\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-kube-api-access-jm9qh\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920472 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920491 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920506 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920520 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aa753f-f902-46e6-9aa6-d64ba51f3472-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920533 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3906bfb-afdb-4752-907f-695607248522-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920547 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.920953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.920913 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:54.921621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.921596 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "kube-api-access-rl2pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.922255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.922233 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:54.922516 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.922494 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:54.922707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:54.922693 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy" pod "b3906bfb-afdb-4752-907f-695607248522" (UID: "b3906bfb-afdb-4752-907f-695607248522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021426 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021468 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3906bfb-afdb-4752-907f-695607248522-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021480 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021489 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021499 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3906bfb-afdb-4752-907f-695607248522-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.021511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.021509 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2pv\" (UniqueName: \"kubernetes.io/projected/b3906bfb-afdb-4752-907f-695607248522-kube-api-access-rl2pv\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:55.483430 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.483397 5209 generic.go:356] "Generic (PLEG): container finished" podID="b3906bfb-afdb-4752-907f-695607248522" containerID="c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb" exitCode=0 Nov 03 19:20:55.483877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.483451 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" Nov 03 19:20:55.483877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.483450 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" event={"ID":"b3906bfb-afdb-4752-907f-695607248522","Type":"ContainerDied","Data":"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb"} Nov 03 19:20:55.483877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.483587 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6" event={"ID":"b3906bfb-afdb-4752-907f-695607248522","Type":"ContainerDied","Data":"3479956040f08eb798b1df436258c1e4af63919ee43c89de1bac1b66fc37b15c"} Nov 03 19:20:55.483877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.483614 5209 scope.go:122] "RemoveContainer" containerID="c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb" Nov 03 19:20:55.485004 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.484984 5209 generic.go:356] "Generic (PLEG): container finished" podID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerID="e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b" exitCode=0 Nov 03 19:20:55.485092 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.485016 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" event={"ID":"38aa753f-f902-46e6-9aa6-d64ba51f3472","Type":"ContainerDied","Data":"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b"} Nov 03 19:20:55.485092 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.485037 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" Nov 03 19:20:55.485092 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.485049 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68" event={"ID":"38aa753f-f902-46e6-9aa6-d64ba51f3472","Type":"ContainerDied","Data":"aca408234c606c6b6c7fe024ba6d504ce4750cd22953fd15efbbda1d1d63a6ce"} Nov 03 19:20:55.495555 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.495535 5209 scope.go:122] "RemoveContainer" containerID="c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb" Nov 03 19:20:55.495868 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:55.495848 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb\": container with ID starting with c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb not found: ID does not exist" containerID="c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb" Nov 03 19:20:55.495930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.495876 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb"} err="failed to get container status \"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb\": rpc error: code = NotFound desc = could not find container \"c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb\": container with ID starting with c2b50e26ee247ce254c793886e855a9f3172523e88c81be578e5d5d170f3aefb not found: ID does not exist" Nov 03 19:20:55.495930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.495895 5209 scope.go:122] "RemoveContainer" containerID="e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b" Nov 03 19:20:55.504684 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.504667 5209 scope.go:122] "RemoveContainer" containerID="e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b" Nov 03 19:20:55.504975 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:55.504957 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b\": container with ID starting with e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b not found: ID does not exist" containerID="e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b" Nov 03 19:20:55.505050 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.504982 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b"} err="failed to get container status \"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b\": rpc error: code = NotFound desc = could not find container \"e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b\": container with ID starting with e2309c057ae7279af19defe23ea4fe10b227a76cb948f739dabde8b80cfc871b not found: ID does not exist" Nov 03 19:20:55.639233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.639200 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:20:55.685331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.685302 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-95l68"] Nov 03 19:20:55.748375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.748302 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:20:55.754446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:55.754418 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-f6tq6"] Nov 03 19:20:56.387184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.385194 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f7b7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:20:56.387184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.385274 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 19:20:56.412152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.412107 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 19:20:56.482261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.482237 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:20:56.495795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.495766 5209 generic.go:356] "Generic (PLEG): container finished" podID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerID="d9b84ffa76726360feb3b558b835e7170c22060abc13023a7a75a977f0684ce8" exitCode=0 Nov 03 19:20:56.496203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.495830 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" event={"ID":"a621c5bb-d630-4187-96b8-1799e249f0ed","Type":"ContainerDied","Data":"d9b84ffa76726360feb3b558b835e7170c22060abc13023a7a75a977f0684ce8"} Nov 03 19:20:56.499039 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.499013 5209 generic.go:356] "Generic (PLEG): container finished" podID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerID="8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85" exitCode=0 Nov 03 19:20:56.499213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.499070 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" Nov 03 19:20:56.499213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.499113 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerDied","Data":"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85"} Nov 03 19:20:56.499213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.499170 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb" event={"ID":"aa5972fb-ca3d-404f-96a2-39f40041cb28","Type":"ContainerDied","Data":"0e97b5e725bbd90bdd65b87bed3a459dddc4dd7145eb174d8648fe22b538cc1c"} Nov 03 19:20:56.499213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.499200 5209 scope.go:122] "RemoveContainer" containerID="8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85" Nov 03 19:20:56.501159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.501099 5209 generic.go:356] "Generic (PLEG): container finished" podID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerID="0bab422f2f553187efc1d45145248f159175a944d32e109044f4dcdf84342ea0" exitCode=0 Nov 03 19:20:56.501297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.501157 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" event={"ID":"b9bd9579-6f86-4854-8129-50996f0db9fa","Type":"ContainerDied","Data":"0bab422f2f553187efc1d45145248f159175a944d32e109044f4dcdf84342ea0"} Nov 03 19:20:56.503787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.503764 5209 generic.go:356] "Generic (PLEG): container finished" podID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerID="f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312" exitCode=0 Nov 03 19:20:56.503880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.503798 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" event={"ID":"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee","Type":"ContainerDied","Data":"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312"} Nov 03 19:20:56.503880 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.503835 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" event={"ID":"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee","Type":"ContainerDied","Data":"524824e6254bdaec4bafd39dd1c54deccd24ecdb80dfb3ea8c8d3f000e78df1f"} Nov 03 19:20:56.503959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.503904 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8" Nov 03 19:20:56.511692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.511670 5209 scope.go:122] "RemoveContainer" containerID="12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7" Nov 03 19:20:56.520860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.520830 5209 scope.go:122] "RemoveContainer" containerID="8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85" Nov 03 19:20:56.521189 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:56.521164 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85\": container with ID starting with 8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85 not found: ID does not exist" containerID="8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85" Nov 03 19:20:56.521268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.521200 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85"} err="failed to get container status \"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85\": rpc error: code = NotFound desc = could not find container \"8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85\": container with ID starting with 8e9f892a16714111d1932000a0863ecb7196465061108cfe4eea1e2b8d376d85 not found: ID does not exist" Nov 03 19:20:56.521268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.521228 5209 scope.go:122] "RemoveContainer" containerID="12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7" Nov 03 19:20:56.521518 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:56.521499 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7\": container with ID starting with 12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7 not found: ID does not exist" containerID="12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7" Nov 03 19:20:56.521589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.521527 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7"} err="failed to get container status \"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7\": rpc error: code = NotFound desc = could not find container \"12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7\": container with ID starting with 12bb43af0003ac9a74ec0be55f382b219bb9cf54216f43d9b5f10798ded00ad7 not found: ID does not exist" Nov 03 19:20:56.521589 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.521552 5209 scope.go:122] "RemoveContainer" containerID="f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312" Nov 03 19:20:56.534051 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.534027 5209 scope.go:122] "RemoveContainer" containerID="f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312" Nov 03 19:20:56.534596 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:20:56.534571 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312\": container with ID starting with f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312 not found: ID does not exist" containerID="f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312" Nov 03 19:20:56.534687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.534602 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312"} err="failed to get container status \"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312\": rpc error: code = NotFound desc = could not find container \"f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312\": container with ID starting with f9f2f4af4bb6279d348832118d3fb9329b704964e98d0c1b49be858675ca3312 not found: ID does not exist" Nov 03 19:20:56.537207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.537184 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:20:56.538920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.538897 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539017 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.538953 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539017 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539009 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539151 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539236 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539273 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539330 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539363 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539396 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\") pod \"aa5972fb-ca3d-404f-96a2-39f40041cb28\" (UID: \"aa5972fb-ca3d-404f-96a2-39f40041cb28\") " Nov 03 19:20:56.539823 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.539795 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.540084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.540057 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.540201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.540172 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:56.540679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.540652 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.540768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.540725 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.542240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.542220 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:56.542571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.542545 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "kube-api-access-m6d2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.542942 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.542922 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.543023 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.542974 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy" pod "aa5972fb-ca3d-404f-96a2-39f40041cb28" (UID: "aa5972fb-ca3d-404f-96a2-39f40041cb28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.550257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.550239 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:20:56.640799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640721 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.640799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640761 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.640799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640798 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640834 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640865 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640919 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.640950 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641016 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641040 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641058 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641090 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641181 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641170 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641226 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641243 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641261 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641283 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641298 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641343 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641379 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641407 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641497 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641512 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\") pod \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\" (UID: \"289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee\") " Nov 03 19:20:56.641557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641560 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\") pod \"a621c5bb-d630-4187-96b8-1799e249f0ed\" (UID: \"a621c5bb-d630-4187-96b8-1799e249f0ed\") " Nov 03 19:20:56.642177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641711 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.642177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641911 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.642177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.641939 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:56.642343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642228 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:56.642343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642277 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642301 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642321 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642335 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642347 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642361 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642373 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642388 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6d2m\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-kube-api-access-m6d2m\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642403 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642418 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642431 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5972fb-ca3d-404f-96a2-39f40041cb28-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642444 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642458 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642473 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642486 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642500 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642514 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.642619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.642528 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5972fb-ca3d-404f-96a2-39f40041cb28-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.643594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.643560 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "kube-api-access-rx2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.643700 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.643660 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.643869 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.643836 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.643869 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.643857 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.644453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.644426 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo" pod "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" (UID: "289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:56.644562 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.644539 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "kube-api-access-xfcfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.645008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.644988 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:56.645095 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.645013 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token" pod "a621c5bb-d630-4187-96b8-1799e249f0ed" (UID: "a621c5bb-d630-4187-96b8-1799e249f0ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.699508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.699479 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" path="/var/lib/kubelet/pods/38aa753f-f902-46e6-9aa6-d64ba51f3472/volumes" Nov 03 19:20:56.699970 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.699957 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3906bfb-afdb-4752-907f-695607248522" path="/var/lib/kubelet/pods/b3906bfb-afdb-4752-907f-695607248522/volumes" Nov 03 19:20:56.743361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743334 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743489 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743378 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743489 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743450 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743489 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743476 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743505 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743562 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743603 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.743806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743662 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743747 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.743806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743772 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.743806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743782 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.743996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743848 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:56.743996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.743864 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\") pod \"b9bd9579-6f86-4854-8129-50996f0db9fa\" (UID: \"b9bd9579-6f86-4854-8129-50996f0db9fa\") " Nov 03 19:20:56.744210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744178 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.744406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744372 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744402 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744417 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a621c5bb-d630-4187-96b8-1799e249f0ed-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744434 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744449 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744464 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744479 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744494 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744510 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2ps\" (UniqueName: \"kubernetes.io/projected/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee-kube-api-access-rx2ps\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744525 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd9579-6f86-4854-8129-50996f0db9fa-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744539 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744553 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.744566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744567 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.745175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.744583 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcfj\" (UniqueName: \"kubernetes.io/projected/a621c5bb-d630-4187-96b8-1799e249f0ed-kube-api-access-xfcfj\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.745813 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.745788 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.745946 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.745927 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "kube-api-access-p8vmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.746091 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.746072 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:56.746192 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.746095 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token" pod "b9bd9579-6f86-4854-8129-50996f0db9fa" (UID: "b9bd9579-6f86-4854-8129-50996f0db9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.846086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.846044 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.846086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.846081 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vmd\" (UniqueName: \"kubernetes.io/projected/b9bd9579-6f86-4854-8129-50996f0db9fa-kube-api-access-p8vmd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.846302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.846098 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.846302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:56.846110 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd9579-6f86-4854-8129-50996f0db9fa-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:57.027750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.027667 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:20:57.035162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.035114 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xg2b8"] Nov 03 19:20:57.138052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.138015 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 19:20:57.148017 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.147894 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hlqmb"] Nov 03 19:20:57.513149 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.513098 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" Nov 03 19:20:57.513594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.513151 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl" event={"ID":"a621c5bb-d630-4187-96b8-1799e249f0ed","Type":"ContainerDied","Data":"e9a96ea3edfefff14c818d7dee76efc96c87544db623367de1cb9713a3a23497"} Nov 03 19:20:57.513594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.513253 5209 scope.go:122] "RemoveContainer" containerID="d9b84ffa76726360feb3b558b835e7170c22060abc13023a7a75a977f0684ce8" Nov 03 19:20:57.516152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.516106 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" Nov 03 19:20:57.516255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.516154 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s" event={"ID":"b9bd9579-6f86-4854-8129-50996f0db9fa","Type":"ContainerDied","Data":"e2851e8cc6f07bf75bc0aa84bef228e784a8fd49288f229d9c6e8a5582fe3fbd"} Nov 03 19:20:57.526819 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.526794 5209 scope.go:122] "RemoveContainer" containerID="0bab422f2f553187efc1d45145248f159175a944d32e109044f4dcdf84342ea0" Nov 03 19:20:57.669332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.669305 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:20:57.673581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.673558 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8vfcl"] Nov 03 19:20:57.776017 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.775987 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:20:57.783707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:57.783679 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f7b7s"] Nov 03 19:20:58.699566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:58.699532 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" path="/var/lib/kubelet/pods/289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee/volumes" Nov 03 19:20:58.700197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:58.700178 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" path="/var/lib/kubelet/pods/a621c5bb-d630-4187-96b8-1799e249f0ed/volumes" Nov 03 19:20:58.700663 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:58.700644 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" path="/var/lib/kubelet/pods/aa5972fb-ca3d-404f-96a2-39f40041cb28/volumes" Nov 03 19:20:58.701469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:20:58.701449 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" path="/var/lib/kubelet/pods/b9bd9579-6f86-4854-8129-50996f0db9fa/volumes" Nov 03 19:21:01.937015 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:21:01.936976 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:21:06.840678 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.840635 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:21:06.841422 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841401 5209 cpu_manager.go:414] "containerMap: removing container" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841423 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841444 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841451 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841468 5209 cpu_manager.go:414] "containerMap: removing container" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841475 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841499 5209 cpu_manager.go:414] "containerMap: removing container" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841506 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841521 5209 cpu_manager.go:414] "containerMap: removing container" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841527 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841543 5209 cpu_manager.go:414] "containerMap: removing container" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.841551 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841549 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841573 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841581 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841592 5209 cpu_manager.go:414] "containerMap: removing container" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841601 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841758 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="a621c5bb-d630-4187-96b8-1799e249f0ed" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841775 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bd9579-6f86-4854-8129-50996f0db9fa" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841788 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841800 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="38aa753f-f902-46e6-9aa6-d64ba51f3472" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841808 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="289f055e-b9a3-4ce7-8e59-db3cbf0fb1ee" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841818 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5972fb-ca3d-404f-96a2-39f40041cb28" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841828 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3906bfb-afdb-4752-907f-695607248522" containerName="istio-proxy" Nov 03 19:21:06.842078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.841836 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42d275d-beb4-4789-861b-83632fbba289" containerName="istio-proxy" Nov 03 19:21:06.847055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.847025 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.896155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.896110 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6413ba-openshift-default-dockercfg-7kgs8\"" Nov 03 19:21:06.947641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.947604 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:21:06.975010 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.974972 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975071 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975104 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsbw\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975217 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975439 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975269 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975439 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975330 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975439 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975416 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975457 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:06.975617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:06.975588 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076681 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076729 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076749 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsbw\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076784 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076830 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.076892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076869 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076922 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.076962 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.077027 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.077205 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.077258 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077471 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.077422 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.077731 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.077685 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.078057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.078037 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.079419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.079393 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.079516 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.079470 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.135069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.134996 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.135274 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.135248 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsbw\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\") pod \"e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.160206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.160178 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:07.597799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.597763 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:21:07.603422 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:21:07.603393 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf367ac2d_ad2d_4edb_85c5_8e32e7e3cde4.slice/crio-69da8a4158802fd875b73b48107a8b4473ef4588add6b422ba911e53e23c4376 WatchSource:0}: Error finding container 69da8a4158802fd875b73b48107a8b4473ef4588add6b422ba911e53e23c4376: Status 404 returned error can't find the container with id 69da8a4158802fd875b73b48107a8b4473ef4588add6b422ba911e53e23c4376 Nov 03 19:21:07.606374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.606344 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:07.606469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.606405 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:07.606469 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:07.606449 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:08.594011 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:08.593975 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerStarted","Data":"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b"} Nov 03 19:21:08.594515 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:08.594064 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerStarted","Data":"69da8a4158802fd875b73b48107a8b4473ef4588add6b422ba911e53e23c4376"} Nov 03 19:21:08.756881 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:08.756819 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podStartSLOduration=2.756806444 podStartE2EDuration="2.756806444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:08.754254193 +0000 UTC m=+2150.165354350" watchObservedRunningTime="2026-11-03 19:21:08.756806444 +0000 UTC m=+2150.167906567" Nov 03 19:21:09.160897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:09.160857 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:09.166761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:09.166734 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:09.600531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:09.600497 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:09.601507 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:09.601486 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:21:15.294969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.294933 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:21:15.298840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.298811 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.383487 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.383443 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.398758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.398723 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399059 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399031 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf889\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399240 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399390 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399509 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399678 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.399937 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.399916 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.400078 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.400065 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.400825 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.400798 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:21:15.401348 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.401321 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" podUID="284b815e-43dc-476b-a473-874262fca5a3" containerName="maas-api" containerID="cri-o://daf00fb2dcf8230b0f6bec0fe1c4c758f6c2124cfb95048c90d637295e1f4655" gracePeriod=30 Nov 03 19:21:15.407177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.407154 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:21:15.408859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.408824 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-332dac-openshift-default-dockercfg-w2jm5\"" Nov 03 19:21:15.507719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.507681 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510412 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf889\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510458 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510504 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510543 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510621 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510738 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510782 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.510845 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.511334 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.511941 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.512157 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.512331 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.513147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.512538 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.525136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.515425 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.525136 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.523636 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.577155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.575875 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.577155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.575932 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf889\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.618717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.618667 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:15.652165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.649362 5209 generic.go:356] "Generic (PLEG): container finished" podID="284b815e-43dc-476b-a473-874262fca5a3" containerID="daf00fb2dcf8230b0f6bec0fe1c4c758f6c2124cfb95048c90d637295e1f4655" exitCode=0 Nov 03 19:21:15.652165 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.649649 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" event={"ID":"284b815e-43dc-476b-a473-874262fca5a3","Type":"ContainerDied","Data":"daf00fb2dcf8230b0f6bec0fe1c4c758f6c2124cfb95048c90d637295e1f4655"} Nov 03 19:21:15.720142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.720104 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:21:15.820578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.820544 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\" (UniqueName: \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\") pod \"284b815e-43dc-476b-a473-874262fca5a3\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " Nov 03 19:21:15.820749 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.820682 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\") pod \"284b815e-43dc-476b-a473-874262fca5a3\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " Nov 03 19:21:15.820820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.820806 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\") pod \"284b815e-43dc-476b-a473-874262fca5a3\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " Nov 03 19:21:15.820869 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.820837 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") pod \"284b815e-43dc-476b-a473-874262fca5a3\" (UID: \"284b815e-43dc-476b-a473-874262fca5a3\") " Nov 03 19:21:15.821327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.821224 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca" pod "284b815e-43dc-476b-a473-874262fca5a3" (UID: "284b815e-43dc-476b-a473-874262fca5a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:15.821477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.821422 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/284b815e-43dc-476b-a473-874262fca5a3-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:15.822877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.822840 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h" pod "284b815e-43dc-476b-a473-874262fca5a3" (UID: "284b815e-43dc-476b-a473-874262fca5a3"). InnerVolumeSpecName "kube-api-access-jt26h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:15.823234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.823208 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls" pod "284b815e-43dc-476b-a473-874262fca5a3" (UID: "284b815e-43dc-476b-a473-874262fca5a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:15.824090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.824065 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs" pod "284b815e-43dc-476b-a473-874262fca5a3" (UID: "284b815e-43dc-476b-a473-874262fca5a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:15.922785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.922706 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:15.922785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.922734 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/284b815e-43dc-476b-a473-874262fca5a3-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:15.922785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:15.922744 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt26h\" (UniqueName: \"kubernetes.io/projected/284b815e-43dc-476b-a473-874262fca5a3-kube-api-access-jt26h\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:16.130803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.130767 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:21:16.131360 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:21:16.131332 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91bb60f_afed_4f87_a192_283e15d7ad31.slice/crio-2c206fef5debacec43e0dbba9350aa651f15a71658cca10f2cc2be9a874cf879 WatchSource:0}: Error finding container 2c206fef5debacec43e0dbba9350aa651f15a71658cca10f2cc2be9a874cf879: Status 404 returned error can't find the container with id 2c206fef5debacec43e0dbba9350aa651f15a71658cca10f2cc2be9a874cf879 Nov 03 19:21:16.134444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.134408 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:16.134518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.134491 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:16.134560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.134537 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:21:16.660135 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.660081 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerStarted","Data":"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7"} Nov 03 19:21:16.660570 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.660165 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerStarted","Data":"2c206fef5debacec43e0dbba9350aa651f15a71658cca10f2cc2be9a874cf879"} Nov 03 19:21:16.661974 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.661953 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" Nov 03 19:21:16.662141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.661997 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" event={"ID":"284b815e-43dc-476b-a473-874262fca5a3","Type":"ContainerDied","Data":"9258788fcf36e721215f58f96ba6e464b53646c30c6babea9f7e06f8c7f0d9f1"} Nov 03 19:21:16.662141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.662071 5209 scope.go:122] "RemoveContainer" containerID="daf00fb2dcf8230b0f6bec0fe1c4c758f6c2124cfb95048c90d637295e1f4655" Nov 03 19:21:16.775404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.775276 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podStartSLOduration=1.775254487 podStartE2EDuration="1.775254487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:16.773614033 +0000 UTC m=+2158.184714181" watchObservedRunningTime="2026-11-03 19:21:16.775254487 +0000 UTC m=+2158.186354611" Nov 03 19:21:16.887141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.886046 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:21:16.895092 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:16.892946 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn"] Nov 03 19:21:17.017638 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.017605 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:21:17.018303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.018283 5209 cpu_manager.go:414] "containerMap: removing container" podUID="284b815e-43dc-476b-a473-874262fca5a3" containerName="maas-api" Nov 03 19:21:17.018303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.018304 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b815e-43dc-476b-a473-874262fca5a3" containerName="maas-api" Nov 03 19:21:17.018420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.018413 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b815e-43dc-476b-a473-874262fca5a3" containerName="maas-api" Nov 03 19:21:17.023144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.023102 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.037526 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.037498 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.037651 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.037559 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbr9k\" (UniqueName: \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.073035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.072993 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6413ba\"" Nov 03 19:21:17.077462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.077437 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:21:17.115305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.115280 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6413ba-dockercfg-mczn6\"" Nov 03 19:21:17.138766 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.138735 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.138904 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.138789 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbr9k\" (UniqueName: \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.139702 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.139675 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.183822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.183788 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:21:17.188506 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.188483 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.202651 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.202619 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbr9k\" (UniqueName: \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\") pod \"payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.240199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.239809 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.240199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.239875 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jjl\" (UniqueName: \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.241793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.241763 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:21:17.341614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.341347 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:17.382502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.362878 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.382502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.362955 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jjl\" (UniqueName: \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.382502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.364290 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.429156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.426768 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jjl\" (UniqueName: \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\") pod \"payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.503367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.503291 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:17.621265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.621190 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:17.627505 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.627479 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:17.670364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.670334 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:17.671226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.671203 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:21:17.797152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.796165 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:21:17.883644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:17.883572 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:21:18.679475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.676625 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08baeb-7b98ddd9bb-48ntn" podUID="284b815e-43dc-476b-a473-874262fca5a3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.89:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:21:18.686021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.685989 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" event={"ID":"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5","Type":"ContainerStarted","Data":"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea"} Nov 03 19:21:18.686219 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.686050 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" event={"ID":"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5","Type":"ContainerStarted","Data":"6e7060d8168df5d729e5de0a8b46e357d8fbb363b936167ed4eb6f135fdfa26d"} Nov 03 19:21:18.686219 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.686082 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:18.688021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.687992 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" event={"ID":"886455ec-0ba7-4333-b783-3606386adf82","Type":"ContainerStarted","Data":"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae"} Nov 03 19:21:18.688235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.688052 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" event={"ID":"886455ec-0ba7-4333-b783-3606386adf82","Type":"ContainerStarted","Data":"5944df31aab60d9dac23792b40394b8e69ef43f93b1e6a067a4fcdea6819391f"} Nov 03 19:21:18.688235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.688087 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:18.700269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.700244 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b815e-43dc-476b-a473-874262fca5a3" path="/var/lib/kubelet/pods/284b815e-43dc-476b-a473-874262fca5a3/volumes" Nov 03 19:21:18.797497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.797434 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" podStartSLOduration=1.797419289 podStartE2EDuration="1.797419289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:18.794620545 +0000 UTC m=+2160.205720680" watchObservedRunningTime="2026-11-03 19:21:18.797419289 +0000 UTC m=+2160.208519412" Nov 03 19:21:18.908041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:18.907984 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" podStartSLOduration=2.907968172 podStartE2EDuration="2.907968172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:18.90693621 +0000 UTC m=+2160.318036332" watchObservedRunningTime="2026-11-03 19:21:18.907968172 +0000 UTC m=+2160.319068296" Nov 03 19:21:25.145495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.145458 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:21:25.151994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.151968 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.203780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.203750 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-332dac-dockercfg-m58h9\"" Nov 03 19:21:25.256142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.250987 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-332dac\"" Nov 03 19:21:25.256142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.256067 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:21:25.268947 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.268907 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2c6g\" (UniqueName: \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.269164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.268990 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.312220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.312180 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:21:25.316605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.316583 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.369799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.369769 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2c6g\" (UniqueName: \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.369948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.369808 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.369948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.369845 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.370064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.369957 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvkq\" (UniqueName: \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.370578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.370555 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.377677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.377656 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:21:25.447065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.446998 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2c6g\" (UniqueName: \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\") pod \"payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.465418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.465391 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:25.473639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.473595 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.473948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.473923 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvkq\" (UniqueName: \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.474893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.474871 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.532639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.532565 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvkq\" (UniqueName: \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\") pod \"payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.631142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.630343 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:25.849720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:25.849691 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:21:25.854740 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:21:25.854717 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873a9b12_181c_49ed_9a18_2e27cc380fd3.slice/crio-fc030007b939960b674c51f5556ae2efec3fbe50d3352ec474554c8ce5f33974 WatchSource:0}: Error finding container fc030007b939960b674c51f5556ae2efec3fbe50d3352ec474554c8ce5f33974: Status 404 returned error can't find the container with id fc030007b939960b674c51f5556ae2efec3fbe50d3352ec474554c8ce5f33974 Nov 03 19:21:26.013833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.013763 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:21:26.018016 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:21:26.017993 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a985a_baa9_426b_b5c5_d4d21a64228a.slice/crio-2d89693c43b2ce3463e2373def35500276936845fed010e1086b550eab88ae09 WatchSource:0}: Error finding container 2d89693c43b2ce3463e2373def35500276936845fed010e1086b550eab88ae09: Status 404 returned error can't find the container with id 2d89693c43b2ce3463e2373def35500276936845fed010e1086b550eab88ae09 Nov 03 19:21:26.748560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.748519 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" event={"ID":"873a9b12-181c-49ed-9a18-2e27cc380fd3","Type":"ContainerStarted","Data":"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e"} Nov 03 19:21:26.749042 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.748587 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:26.749042 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.748612 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" event={"ID":"873a9b12-181c-49ed-9a18-2e27cc380fd3","Type":"ContainerStarted","Data":"fc030007b939960b674c51f5556ae2efec3fbe50d3352ec474554c8ce5f33974"} Nov 03 19:21:26.750168 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.750138 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" event={"ID":"2a0a985a-baa9-426b-b5c5-d4d21a64228a","Type":"ContainerStarted","Data":"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d"} Nov 03 19:21:26.750304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.750193 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" event={"ID":"2a0a985a-baa9-426b-b5c5-d4d21a64228a","Type":"ContainerStarted","Data":"2d89693c43b2ce3463e2373def35500276936845fed010e1086b550eab88ae09"} Nov 03 19:21:26.750304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.750238 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:26.867579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:26.867526 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" podStartSLOduration=1.867511732 podStartE2EDuration="1.867511732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:26.864401066 +0000 UTC m=+2168.275501207" watchObservedRunningTime="2026-11-03 19:21:26.867511732 +0000 UTC m=+2168.278612096" Nov 03 19:21:27.007305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:27.007191 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" podStartSLOduration=2.007174786 podStartE2EDuration="2.007174786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:27.006227128 +0000 UTC m=+2168.417327279" watchObservedRunningTime="2026-11-03 19:21:27.007174786 +0000 UTC m=+2168.418274910" Nov 03 19:21:29.701781 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:29.701696 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:21:29.701781 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:29.701766 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:21:31.937844 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:21:31.937807 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:21:37.759851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:37.759823 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:21:37.760298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:37.759898 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:21:49.719360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:21:49.719328 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:22:00.038901 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.038867 5209 generic.go:356] "Generic (PLEG): container finished" podID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerID="99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8" exitCode=0 Nov 03 19:22:00.039345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.038944 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerDied","Data":"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8"} Nov 03 19:22:00.039504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.039482 5209 scope.go:122] "RemoveContainer" containerID="99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8" Nov 03 19:22:00.041213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.041177 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:22:00.041306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.041249 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:22:00.041306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.041280 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:22:00.083846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:00.083826 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:22:01.047155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:01.047108 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerStarted","Data":"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512"} Nov 03 19:22:01.101248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:01.101205 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:22:01.102067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:01.102040 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:01.102227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:01.102095 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:01.937923 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:22:01.937887 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:22:02.101518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:02.101479 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:02.101912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:02.101551 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:03.101508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:03.101466 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:03.101693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:03.101539 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:04.101674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:04.101627 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:04.101883 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:04.101700 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:05.101466 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:05.101432 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:05.101687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:05.101482 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:06.101542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:06.101503 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:22:06.102068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:06.101571 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 19:22:07.105594 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:07.105558 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:22:07.106081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:07.106050 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:22:07.106721 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:07.106705 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:22:31.552303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:31.552266 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:22:31.937247 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:22:31.937211 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:22:59.260998 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.260921 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8863-0020d9"] Nov 03 19:22:59.265709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.265686 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:22:59.306576 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.306541 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8863-0020d9"] Nov 03 19:22:59.349566 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.349527 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2f8z4\"" Nov 03 19:22:59.357740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.357704 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcztf\" (UniqueName: \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\") pod \"test-curl-8863-0020d9\" (UID: \"4a455f95-3601-4ffb-abd1-806f1cfec39c\") " pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:22:59.459256 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.459224 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcztf\" (UniqueName: \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\") pod \"test-curl-8863-0020d9\" (UID: \"4a455f95-3601-4ffb-abd1-806f1cfec39c\") " pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:22:59.516707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.516632 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcztf\" (UniqueName: \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\") pod \"test-curl-8863-0020d9\" (UID: \"4a455f95-3601-4ffb-abd1-806f1cfec39c\") " pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:22:59.583840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.583808 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:22:59.957285 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:22:59.957255 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8863-0020d9"] Nov 03 19:22:59.959434 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:22:59.959403 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a455f95_3601_4ffb_abd1_806f1cfec39c.slice/crio-675c5dde299625ed700f950ab61a7d43f3f3bc75f91ce9c25fb42c8be338a993 WatchSource:0}: Error finding container 675c5dde299625ed700f950ab61a7d43f3f3bc75f91ce9c25fb42c8be338a993: Status 404 returned error can't find the container with id 675c5dde299625ed700f950ab61a7d43f3f3bc75f91ce9c25fb42c8be338a993 Nov 03 19:23:00.464612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:00.464576 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8863-0020d9" event={"ID":"4a455f95-3601-4ffb-abd1-806f1cfec39c","Type":"ContainerStarted","Data":"341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292"} Nov 03 19:23:00.465026 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:00.464623 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8863-0020d9" event={"ID":"4a455f95-3601-4ffb-abd1-806f1cfec39c","Type":"ContainerStarted","Data":"675c5dde299625ed700f950ab61a7d43f3f3bc75f91ce9c25fb42c8be338a993"} Nov 03 19:23:00.572587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:00.572533 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8863-0020d9" podStartSLOduration=1.572520392 podStartE2EDuration="1.572520392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:00.570105638 +0000 UTC m=+2261.981205763" watchObservedRunningTime="2026-11-03 19:23:00.572520392 +0000 UTC m=+2261.983620514" Nov 03 19:23:01.956954 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:01.956913 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:23:02.144731 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:02.144692 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8863-0020d9"] Nov 03 19:23:02.149743 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:02.149712 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8863-0020d9"] Nov 03 19:23:02.518389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:02.518319 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8863-0020d9" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" containerName="test-curl-8863-0020d9" containerID="cri-o://341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292" gracePeriod=2 Nov 03 19:23:02.566800 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:02.566761 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-0020d9\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:23:05.166726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.166704 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:23:05.205591 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.204658 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\" (UniqueName: \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\") pod \"4a455f95-3601-4ffb-abd1-806f1cfec39c\" (UID: \"4a455f95-3601-4ffb-abd1-806f1cfec39c\") " Nov 03 19:23:05.236890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.236793 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf" pod "4a455f95-3601-4ffb-abd1-806f1cfec39c" (UID: "4a455f95-3601-4ffb-abd1-806f1cfec39c"). InnerVolumeSpecName "kube-api-access-dcztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:05.307112 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.307067 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcztf\" (UniqueName: \"kubernetes.io/projected/4a455f95-3601-4ffb-abd1-806f1cfec39c-kube-api-access-dcztf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:05.630146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.616664 5209 generic.go:356] "Generic (PLEG): container finished" podID="4a455f95-3601-4ffb-abd1-806f1cfec39c" containerID="341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292" exitCode=137 Nov 03 19:23:05.630146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.616895 5209 scope.go:122] "RemoveContainer" containerID="341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292" Nov 03 19:23:05.630146 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.617048 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:23:05.649212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.649190 5209 scope.go:122] "RemoveContainer" containerID="341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292" Nov 03 19:23:05.649490 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:05.649468 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292\": container with ID starting with 341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292 not found: ID does not exist" containerID="341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292" Nov 03 19:23:05.649577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:05.649497 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292"} err="failed to get container status \"341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292\": rpc error: code = NotFound desc = could not find container \"341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292\": container with ID starting with 341e7fe2dbe38b8a08d612afbcb31f269d47da6b9a780e66a4447bde07b57292 not found: ID does not exist" Nov 03 19:23:05.698144 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:05.695401 5209 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-0020d9\" is forbidden: User \"system:node:ip-10-0-133-236.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-236.us-west-2.compute.internal' and this object" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" pod="opendatahub/test-curl-8863-0020d9" Nov 03 19:23:06.715980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:06.715946 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" path="/var/lib/kubelet/pods/4a455f95-3601-4ffb-abd1-806f1cfec39c/volumes" Nov 03 19:23:09.848354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:09.848320 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:23:09.894805 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:09.894766 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 19:23:10.247784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.246928 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": read tcp 10.133.0.2:56792->10.133.0.106:15021: read: connection reset by peer" start-of-body= Nov 03 19:23:10.247784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.247006 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": read tcp 10.133.0.2:56792->10.133.0.106:15021: read: connection reset by peer" Nov 03 19:23:10.671407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.671382 5209 generic.go:356] "Generic (PLEG): container finished" podID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerID="110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7" exitCode=0 Nov 03 19:23:10.671549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.671457 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerDied","Data":"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7"} Nov 03 19:23:10.671896 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.671876 5209 scope.go:122] "RemoveContainer" containerID="110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7" Nov 03 19:23:10.673247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673219 5209 generic.go:356] "Generic (PLEG): container finished" podID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerID="a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b" exitCode=0 Nov 03 19:23:10.673355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673294 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerDied","Data":"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b"} Nov 03 19:23:10.673820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673562 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:10.673820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673628 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:10.673820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673662 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:10.673820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.673695 5209 scope.go:122] "RemoveContainer" containerID="a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b" Nov 03 19:23:10.675293 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.675265 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:10.675358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.675330 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:10.675393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:10.675369 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:23:11.729066 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:11.729023 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerStarted","Data":"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e"} Nov 03 19:23:11.731592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:11.731565 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerStarted","Data":"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e"} Nov 03 19:23:12.161035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.161003 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:23:12.161237 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.161204 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:23:12.161285 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.161249 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 19:23:12.625141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.624495 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:23:12.625141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.624660 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:23:12.625141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:12.624704 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 19:23:13.161261 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.161225 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:23:13.161754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.161286 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 19:23:13.404144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.394591 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:23:13.527944 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.527173 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:23:13.621590 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.621552 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:23:13.621909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:13.621861 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 19:23:14.160523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:14.160488 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:23:14.160712 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:14.160541 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 19:23:14.626141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:14.626025 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:23:14.626141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:14.626083 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 19:23:15.160890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:15.160851 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:23:15.161052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:15.160924 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 19:23:15.623145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:15.619450 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:23:15.623145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:15.619515 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 19:23:16.160788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:16.160754 5209 patch_prober.go:29] interesting pod/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:23:16.161211 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:16.160806 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 19:23:16.619502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:16.619462 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:23:16.619813 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:16.619782 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 19:23:17.280221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.280180 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:23:17.281308 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.281285 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:23:17.282280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.282262 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:23:17.624169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.624137 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:23:17.624619 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.624603 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:23:17.625343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:17.625324 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:23:18.394909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.394870 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" podUID="c42ae4f9-4c74-46a0-a076-65980290888c" containerName="payload-processing" containerID="cri-o://969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc" gracePeriod=25 Nov 03 19:23:18.460175 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:18.460141 5209 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42ae4f9_4c74_46a0_a076_65980290888c.slice/crio-969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:23:18.527864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.527808 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" podUID="fd82e045-4677-4ffb-8e62-52d4b6973595" containerName="payload-pre-processing" containerID="cri-o://f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c" gracePeriod=25 Nov 03 19:23:18.658132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.658098 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:23:18.761922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.761891 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\") pod \"c42ae4f9-4c74-46a0-a076-65980290888c\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " Nov 03 19:23:18.762070 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.761988 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\" (UniqueName: \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\") pod \"c42ae4f9-4c74-46a0-a076-65980290888c\" (UID: \"c42ae4f9-4c74-46a0-a076-65980290888c\") " Nov 03 19:23:18.762414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.762229 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume" pod "c42ae4f9-4c74-46a0-a076-65980290888c" (UID: "c42ae4f9-4c74-46a0-a076-65980290888c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:18.762732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.762711 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42ae4f9-4c74-46a0-a076-65980290888c-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:18.764202 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.764176 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2" pod "c42ae4f9-4c74-46a0-a076-65980290888c" (UID: "c42ae4f9-4c74-46a0-a076-65980290888c"). InnerVolumeSpecName "kube-api-access-c88x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:18.772856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.772835 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:23:18.797787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.797750 5209 generic.go:356] "Generic (PLEG): container finished" podID="c42ae4f9-4c74-46a0-a076-65980290888c" containerID="969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc" exitCode=0 Nov 03 19:23:18.797787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.797789 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" Nov 03 19:23:18.798008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.797825 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" event={"ID":"c42ae4f9-4c74-46a0-a076-65980290888c","Type":"ContainerDied","Data":"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc"} Nov 03 19:23:18.798008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.797883 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf" event={"ID":"c42ae4f9-4c74-46a0-a076-65980290888c","Type":"ContainerDied","Data":"266a086418817dff96794010c0b067a264862a5ccf7ed6d668f5442dc777bf3e"} Nov 03 19:23:18.798008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.797910 5209 scope.go:122] "RemoveContainer" containerID="969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc" Nov 03 19:23:18.799802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.799776 5209 generic.go:356] "Generic (PLEG): container finished" podID="fd82e045-4677-4ffb-8e62-52d4b6973595" containerID="f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c" exitCode=0 Nov 03 19:23:18.800247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.799879 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" Nov 03 19:23:18.800247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.799923 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" event={"ID":"fd82e045-4677-4ffb-8e62-52d4b6973595","Type":"ContainerDied","Data":"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c"} Nov 03 19:23:18.800247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.799966 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl" event={"ID":"fd82e045-4677-4ffb-8e62-52d4b6973595","Type":"ContainerDied","Data":"c1f252cec6d8622b932e1a00f361933c7790b5a495db399aa3f603b7dcad1ad7"} Nov 03 19:23:18.809804 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.809787 5209 scope.go:122] "RemoveContainer" containerID="969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc" Nov 03 19:23:18.810133 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:18.810098 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc\": container with ID starting with 969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc not found: ID does not exist" containerID="969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc" Nov 03 19:23:18.810209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.810139 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc"} err="failed to get container status \"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc\": rpc error: code = NotFound desc = could not find container \"969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc\": container with ID starting with 969b62dcba58befe1bdd748b8e09a6f4f845cefe0f404d7b2c0307f0429a9ffc not found: ID does not exist" Nov 03 19:23:18.810209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.810163 5209 scope.go:122] "RemoveContainer" containerID="f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c" Nov 03 19:23:18.821030 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.821015 5209 scope.go:122] "RemoveContainer" containerID="f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c" Nov 03 19:23:18.821313 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:18.821283 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c\": container with ID starting with f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c not found: ID does not exist" containerID="f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c" Nov 03 19:23:18.821397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.821329 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c"} err="failed to get container status \"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c\": rpc error: code = NotFound desc = could not find container \"f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c\": container with ID starting with f652cef8bf05f2cd750a9a785ded955819ff3123bfad4a4a005880a5c38e797c not found: ID does not exist" Nov 03 19:23:18.864075 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.864053 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\" (UniqueName: \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\") pod \"fd82e045-4677-4ffb-8e62-52d4b6973595\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " Nov 03 19:23:18.864283 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.864265 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\") pod \"fd82e045-4677-4ffb-8e62-52d4b6973595\" (UID: \"fd82e045-4677-4ffb-8e62-52d4b6973595\") " Nov 03 19:23:18.864648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.864623 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume" pod "fd82e045-4677-4ffb-8e62-52d4b6973595" (UID: "fd82e045-4677-4ffb-8e62-52d4b6973595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:18.865003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.864979 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd82e045-4677-4ffb-8e62-52d4b6973595-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:18.865100 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.865008 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88x2\" (UniqueName: \"kubernetes.io/projected/c42ae4f9-4c74-46a0-a076-65980290888c-kube-api-access-c88x2\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:18.866183 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.866166 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz" pod "fd82e045-4677-4ffb-8e62-52d4b6973595" (UID: "fd82e045-4677-4ffb-8e62-52d4b6973595"). InnerVolumeSpecName "kube-api-access-6wwxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:18.966043 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:18.965966 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwxz\" (UniqueName: \"kubernetes.io/projected/fd82e045-4677-4ffb-8e62-52d4b6973595-kube-api-access-6wwxz\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:19.041622 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:19.041578 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:23:19.045353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:19.045323 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-nvzhf"] Nov 03 19:23:19.215210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:19.215173 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:23:19.217973 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:19.217916 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-8cttl"] Nov 03 19:23:20.699258 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:20.699227 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42ae4f9-4c74-46a0-a076-65980290888c" path="/var/lib/kubelet/pods/c42ae4f9-4c74-46a0-a076-65980290888c/volumes" Nov 03 19:23:20.699645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:20.699613 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd82e045-4677-4ffb-8e62-52d4b6973595" path="/var/lib/kubelet/pods/fd82e045-4677-4ffb-8e62-52d4b6973595/volumes" Nov 03 19:23:24.870268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:24.870227 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" containerID="cri-o://c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d" gracePeriod=45 Nov 03 19:23:24.915094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:24.915057 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" containerID="cri-o://d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9" gracePeriod=45 Nov 03 19:23:25.193271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.193241 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:23:25.196548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.196527 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:23:25.339057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339021 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339081 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339161 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339188 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339208 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\" (UniqueName: \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339268 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339310 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339365 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\" (UniqueName: \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339412 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339441 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\" (UniqueName: \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339553 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\") pod \"883e3469-c996-4d7c-a954-bb81d753e409\" (UID: \"883e3469-c996-4d7c-a954-bb81d753e409\") " Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339591 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\") pod \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\" (UID: \"d7df4092-2d8b-4989-9b44-9a8ef7aa972a\") " Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339385 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339461 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339593 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339628 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339651 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.339807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.339760 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340234 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340258 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340272 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340286 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340300 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.340335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.340312 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.342481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.342453 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.342608 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.342558 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.342608 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.342569 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:25.342739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.342621 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z" pod "d7df4092-2d8b-4989-9b44-9a8ef7aa972a" (UID: "d7df4092-2d8b-4989-9b44-9a8ef7aa972a"). InnerVolumeSpecName "kube-api-access-b5w4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:25.342918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.342900 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "kube-api-access-zl4f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:25.343041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.343022 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs" pod "883e3469-c996-4d7c-a954-bb81d753e409" (UID: "883e3469-c996-4d7c-a954-bb81d753e409"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:25.442172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442071 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.442172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442108 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883e3469-c996-4d7c-a954-bb81d753e409-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.442172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442141 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4f7\" (UniqueName: \"kubernetes.io/projected/883e3469-c996-4d7c-a954-bb81d753e409-kube-api-access-zl4f7\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.442172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442156 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5w4z\" (UniqueName: \"kubernetes.io/projected/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-kube-api-access-b5w4z\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.442172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442171 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883e3469-c996-4d7c-a954-bb81d753e409-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.442554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.442184 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7df4092-2d8b-4989-9b44-9a8ef7aa972a-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.848044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.848006 5209 generic.go:356] "Generic (PLEG): container finished" podID="883e3469-c996-4d7c-a954-bb81d753e409" containerID="d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9" exitCode=0 Nov 03 19:23:25.848252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.848068 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" Nov 03 19:23:25.848252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.848091 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" event={"ID":"883e3469-c996-4d7c-a954-bb81d753e409","Type":"ContainerDied","Data":"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9"} Nov 03 19:23:25.848252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.848157 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr" event={"ID":"883e3469-c996-4d7c-a954-bb81d753e409","Type":"ContainerDied","Data":"90f59f1b714ec5e359ab3411b050ab02bae1b23994adb4475bb62f75dc05a829"} Nov 03 19:23:25.848252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.848175 5209 scope.go:122] "RemoveContainer" containerID="d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9" Nov 03 19:23:25.849579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.849554 5209 generic.go:356] "Generic (PLEG): container finished" podID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerID="c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d" exitCode=0 Nov 03 19:23:25.849669 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.849626 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" Nov 03 19:23:25.849728 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.849679 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" event={"ID":"d7df4092-2d8b-4989-9b44-9a8ef7aa972a","Type":"ContainerDied","Data":"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d"} Nov 03 19:23:25.849728 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.849704 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9" event={"ID":"d7df4092-2d8b-4989-9b44-9a8ef7aa972a","Type":"ContainerDied","Data":"58f704823a14356a617a655820d62832407074f31fb47489457bc69e8a72baa1"} Nov 03 19:23:25.860019 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.859997 5209 scope.go:122] "RemoveContainer" containerID="d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9" Nov 03 19:23:25.860326 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:25.860303 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9\": container with ID starting with d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9 not found: ID does not exist" containerID="d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9" Nov 03 19:23:25.860407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.860337 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9"} err="failed to get container status \"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9\": rpc error: code = NotFound desc = could not find container \"d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9\": container with ID starting with d1a7d19400d445f4df45dbee5d6d9ed3acef9bfbd4396cf2b6e0f47c837b83e9 not found: ID does not exist" Nov 03 19:23:25.860407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.860359 5209 scope.go:122] "RemoveContainer" containerID="c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d" Nov 03 19:23:25.869150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.869131 5209 scope.go:122] "RemoveContainer" containerID="c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d" Nov 03 19:23:25.869414 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:25.869394 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d\": container with ID starting with c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d not found: ID does not exist" containerID="c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d" Nov 03 19:23:25.869481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:25.869419 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d"} err="failed to get container status \"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d\": rpc error: code = NotFound desc = could not find container \"c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d\": container with ID starting with c6f58de1d8dfb5468bc28b5419eee26228e14ef567659fa1b6d951ba63a1757d not found: ID does not exist" Nov 03 19:23:26.003306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.003274 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 19:23:26.007147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.007108 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/simulator-w4-98860e-kserve-5f4d796567-xjbnr"] Nov 03 19:23:26.110645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.110563 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:23:26.114191 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.114165 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/premium-w4-98860e-kserve-7f89bc8858-b8dn9"] Nov 03 19:23:26.699275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.699244 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883e3469-c996-4d7c-a954-bb81d753e409" path="/var/lib/kubelet/pods/883e3469-c996-4d7c-a954-bb81d753e409/volumes" Nov 03 19:23:26.699692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:26.699679 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" path="/var/lib/kubelet/pods/d7df4092-2d8b-4989-9b44-9a8ef7aa972a/volumes" Nov 03 19:23:31.937444 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:23:31.937405 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:23:38.115863 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:38.115819 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:23:45.597403 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:45.597372 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:23:45.643329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:23:45.643293 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:24:00.618396 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:00.618333 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" containerID="cri-o://ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53" gracePeriod=45 Nov 03 19:24:00.663155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:00.663087 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" containerID="cri-o://a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871" gracePeriod=45 Nov 03 19:24:00.951291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:00.951264 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:24:00.963414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:00.963386 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:24:01.066079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066048 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.066330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066093 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\" (UniqueName: \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066257 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066304 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066382 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.066549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066422 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066549 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066518 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.066703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066607 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.066703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066645 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.066703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066675 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066708 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\") pod \"5559d644-3b16-4c69-9c9f-58447c6442bc\" (UID: \"5559d644-3b16-4c69-9c9f-58447c6442bc\") " Nov 03 19:24:01.066857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.066742 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\" (UniqueName: \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\") pod \"89319f39-d4e0-4080-afdb-350f97b3a9bc\" (UID: \"89319f39-d4e0-4080-afdb-350f97b3a9bc\") " Nov 03 19:24:01.067319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.067287 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.068159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.068082 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.068271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.068178 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.068876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.068627 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.069190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.069167 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.069407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.069385 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.069788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.069754 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:01.070458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.070433 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:01.070585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.070561 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "kube-api-access-mcd8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:01.071399 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.071370 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.071605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.071582 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8" pod "5559d644-3b16-4c69-9c9f-58447c6442bc" (UID: "5559d644-3b16-4c69-9c9f-58447c6442bc"). InnerVolumeSpecName "kube-api-access-rpqj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:01.071851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.071830 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm" pod "89319f39-d4e0-4080-afdb-350f97b3a9bc" (UID: "89319f39-d4e0-4080-afdb-350f97b3a9bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.100710 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.100661 5209 generic.go:356] "Generic (PLEG): container finished" podID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerID="ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53" exitCode=0 Nov 03 19:24:01.100961 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.100708 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" event={"ID":"89319f39-d4e0-4080-afdb-350f97b3a9bc","Type":"ContainerDied","Data":"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53"} Nov 03 19:24:01.101081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.100729 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" Nov 03 19:24:01.101081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.100990 5209 scope.go:122] "RemoveContainer" containerID="ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53" Nov 03 19:24:01.101081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.100977 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5" event={"ID":"89319f39-d4e0-4080-afdb-350f97b3a9bc","Type":"ContainerDied","Data":"6046aad54dc05f675d0d3bbc87d3b5fa945a1ce9bc1a1f482c8be72478774ec8"} Nov 03 19:24:01.102529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.102505 5209 generic.go:356] "Generic (PLEG): container finished" podID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerID="a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871" exitCode=0 Nov 03 19:24:01.102624 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.102539 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" Nov 03 19:24:01.102693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.102676 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" event={"ID":"5559d644-3b16-4c69-9c9f-58447c6442bc","Type":"ContainerDied","Data":"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871"} Nov 03 19:24:01.102765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.102708 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c" event={"ID":"5559d644-3b16-4c69-9c9f-58447c6442bc","Type":"ContainerDied","Data":"d4aead6245ffecc91eb1b722bd1334ed1523df66c2b45e88ad6153cf18b9c2d8"} Nov 03 19:24:01.113939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.113919 5209 scope.go:122] "RemoveContainer" containerID="ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53" Nov 03 19:24:01.114253 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:24:01.114232 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53\": container with ID starting with ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53 not found: ID does not exist" containerID="ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53" Nov 03 19:24:01.114335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.114261 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53"} err="failed to get container status \"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53\": rpc error: code = NotFound desc = could not find container \"ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53\": container with ID starting with ff0baab47ec2a4a5618db340b4ce4464a81c7308bec960d167296ab32bb3ae53 not found: ID does not exist" Nov 03 19:24:01.114335 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.114284 5209 scope.go:122] "RemoveContainer" containerID="a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871" Nov 03 19:24:01.126240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.126210 5209 scope.go:122] "RemoveContainer" containerID="a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871" Nov 03 19:24:01.126718 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:24:01.126633 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871\": container with ID starting with a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871 not found: ID does not exist" containerID="a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871" Nov 03 19:24:01.126718 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.126671 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871"} err="failed to get container status \"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871\": rpc error: code = NotFound desc = could not find container \"a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871\": container with ID starting with a9d9a87b9bbbcf5f0bbbb959b31000d7b74f0d35b72c4f5ba5e70e0065dff871 not found: ID does not exist" Nov 03 19:24:01.167875 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167851 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5559d644-3b16-4c69-9c9f-58447c6442bc-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167893 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167902 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167910 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167919 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167928 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89319f39-d4e0-4080-afdb-350f97b3a9bc-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167935 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167943 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167952 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcd8g\" (UniqueName: \"kubernetes.io/projected/89319f39-d4e0-4080-afdb-350f97b3a9bc-kube-api-access-mcd8g\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167961 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89319f39-d4e0-4080-afdb-350f97b3a9bc-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.167969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167969 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqj8\" (UniqueName: \"kubernetes.io/projected/5559d644-3b16-4c69-9c9f-58447c6442bc-kube-api-access-rpqj8\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.168315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.167978 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5559d644-3b16-4c69-9c9f-58447c6442bc-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.276913 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.276836 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:24:01.279732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.279707 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/premium-w5-90a9bf-kserve-6bd89b5596-5rwv5"] Nov 03 19:24:01.387453 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.387408 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:24:01.393438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:01.393410 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/simulator-w5-90a9bf-kserve-648b5d68d9-bj26c"] Nov 03 19:24:01.937701 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:24:01.937668 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:24:02.698847 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:02.698803 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" path="/var/lib/kubelet/pods/5559d644-3b16-4c69-9c9f-58447c6442bc/volumes" Nov 03 19:24:02.699332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:02.699310 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" path="/var/lib/kubelet/pods/89319f39-d4e0-4080-afdb-350f97b3a9bc/volumes" Nov 03 19:24:08.198131 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198082 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:24:08.198803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198781 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" Nov 03 19:24:08.198803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198804 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198818 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" containerName="test-curl-8863-0020d9" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198823 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" containerName="test-curl-8863-0020d9" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198837 5209 cpu_manager.go:414] "containerMap: removing container" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198842 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198852 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c42ae4f9-4c74-46a0-a076-65980290888c" containerName="payload-processing" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198857 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42ae4f9-4c74-46a0-a076-65980290888c" containerName="payload-processing" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198871 5209 cpu_manager.go:414] "containerMap: removing container" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198876 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198893 5209 cpu_manager.go:414] "containerMap: removing container" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" Nov 03 19:24:08.198910 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198899 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198915 5209 cpu_manager.go:414] "containerMap: removing container" podUID="fd82e045-4677-4ffb-8e62-52d4b6973595" containerName="payload-pre-processing" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198920 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd82e045-4677-4ffb-8e62-52d4b6973595" containerName="payload-pre-processing" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.198999 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a455f95-3601-4ffb-abd1-806f1cfec39c" containerName="test-curl-8863-0020d9" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199009 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="883e3469-c996-4d7c-a954-bb81d753e409" containerName="main" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199015 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42ae4f9-4c74-46a0-a076-65980290888c" containerName="payload-processing" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199022 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7df4092-2d8b-4989-9b44-9a8ef7aa972a" containerName="main" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199030 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="89319f39-d4e0-4080-afdb-350f97b3a9bc" containerName="main" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199038 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd82e045-4677-4ffb-8e62-52d4b6973595" containerName="payload-pre-processing" Nov 03 19:24:08.199248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.199044 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5559d644-3b16-4c69-9c9f-58447c6442bc" containerName="main" Nov 03 19:24:08.203926 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.203907 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.253627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.253595 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.253809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.253716 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlgq\" (UniqueName: \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.255612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.255595 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c3e530\"" Nov 03 19:24:08.301271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.301236 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c3e530-dockercfg-j7679\"" Nov 03 19:24:08.305113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.305077 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:24:08.354553 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.354518 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.354768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.354613 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlgq\" (UniqueName: \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.355452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.355422 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.412730 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.412702 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlgq\" (UniqueName: \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\") pod \"payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.518389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.518310 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:08.895609 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:08.895580 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:24:08.900043 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:24:08.900017 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc7f747_2e30_4cb4_925a_a16ad1167144.slice/crio-d80b27707348f35d557ba12a78088148376611f859de7371504f526bef7bf80b WatchSource:0}: Error finding container d80b27707348f35d557ba12a78088148376611f859de7371504f526bef7bf80b: Status 404 returned error can't find the container with id d80b27707348f35d557ba12a78088148376611f859de7371504f526bef7bf80b Nov 03 19:24:09.165084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:09.164995 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" event={"ID":"ffc7f747-2e30-4cb4-925a-a16ad1167144","Type":"ContainerStarted","Data":"2cc0dda58517da59665cac3ffec383c81e1070694568c777b2906c62212fc8b5"} Nov 03 19:24:09.165084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:09.165078 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" event={"ID":"ffc7f747-2e30-4cb4-925a-a16ad1167144","Type":"ContainerStarted","Data":"d80b27707348f35d557ba12a78088148376611f859de7371504f526bef7bf80b"} Nov 03 19:24:09.165308 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:09.165113 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:09.274450 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:09.274391 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" podStartSLOduration=1.2743764340000001 podStartE2EDuration="1.274376434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:09.273310009 +0000 UTC m=+2330.684410131" watchObservedRunningTime="2026-11-03 19:24:09.274376434 +0000 UTC m=+2330.685476579" Nov 03 19:24:16.077152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:16.076715 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:24:20.174443 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:20.174412 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:24:31.124523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.124407 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerName="main" containerID="cri-o://28055bc983ddbd0c4cc3c44f384eb9b1c9506c5beed88bddb02f37f8a50c796a" gracePeriod=45 Nov 03 19:24:31.314954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.314920 5209 generic.go:356] "Generic (PLEG): container finished" podID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerID="28055bc983ddbd0c4cc3c44f384eb9b1c9506c5beed88bddb02f37f8a50c796a" exitCode=0 Nov 03 19:24:31.315141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.314994 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" event={"ID":"fd1d57f6-7961-4c50-8656-4bb58d85db96","Type":"ContainerDied","Data":"28055bc983ddbd0c4cc3c44f384eb9b1c9506c5beed88bddb02f37f8a50c796a"} Nov 03 19:24:31.390230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.390170 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:24:31.445243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445213 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445256 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445287 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445335 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445395 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445472 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:31.445636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445511 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\" (UniqueName: \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\") pod \"fd1d57f6-7961-4c50-8656-4bb58d85db96\" (UID: \"fd1d57f6-7961-4c50-8656-4bb58d85db96\") " Nov 03 19:24:31.445636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445536 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:31.445764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.445727 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:31.446031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.446007 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.446115 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.446032 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.446115 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.446041 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.447612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.447588 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:31.447736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.447607 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:31.447736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.447618 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9" pod "fd1d57f6-7961-4c50-8656-4bb58d85db96" (UID: "fd1d57f6-7961-4c50-8656-4bb58d85db96"). InnerVolumeSpecName "kube-api-access-8h5z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:31.547523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.547493 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5z9\" (UniqueName: \"kubernetes.io/projected/fd1d57f6-7961-4c50-8656-4bb58d85db96-kube-api-access-8h5z9\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.547523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.547520 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd1d57f6-7961-4c50-8656-4bb58d85db96-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.547523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:31.547529 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd1d57f6-7961-4c50-8656-4bb58d85db96-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:31.937418 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:24:31.937382 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:24:32.322316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.322230 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" Nov 03 19:24:32.322316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.322262 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm" event={"ID":"fd1d57f6-7961-4c50-8656-4bb58d85db96","Type":"ContainerDied","Data":"f0fc07553295208d8c1048db0eca9ba049fbe1de79ae227a05e0ca9e1a76994a"} Nov 03 19:24:32.322788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.322321 5209 scope.go:122] "RemoveContainer" containerID="28055bc983ddbd0c4cc3c44f384eb9b1c9506c5beed88bddb02f37f8a50c796a" Nov 03 19:24:32.477068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.477034 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:24:32.481044 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.481020 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d86d1cd0/e2e-lifecycle-model-d86d1cd0-kserve-5877f48f69-njfrm"] Nov 03 19:24:32.699454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:32.699425 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" path="/var/lib/kubelet/pods/fd1d57f6-7961-4c50-8656-4bb58d85db96/volumes" Nov 03 19:24:41.055056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:41.055016 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:24:55.405992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.405955 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:24:55.406640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.406566 5209 cpu_manager.go:414] "containerMap: removing container" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerName="main" Nov 03 19:24:55.406640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.406581 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerName="main" Nov 03 19:24:55.406765 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.406705 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1d57f6-7961-4c50-8656-4bb58d85db96" containerName="main" Nov 03 19:24:55.409902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.409883 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.459037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.458994 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1410ed\"/\"kube-root-ca.crt\"" Nov 03 19:24:55.505535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.505497 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1410ed\"/\"test-model-1410ed-kserve-self-signed-certs\"" Nov 03 19:24:55.505535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.505497 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1410ed\"/\"default-dockercfg-s9k5q\"" Nov 03 19:24:55.508357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.508334 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:24:55.541166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541134 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.541286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541257 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljgf\" (UniqueName: \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.541339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541284 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.541339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541316 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.541467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541453 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.541531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.541517 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.562658 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.562600 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1410ed\"/\"openshift-service-ca.crt\"" Nov 03 19:24:55.642221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642188 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642257 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642329 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljgf\" (UniqueName: \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642357 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642390 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642484 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642858 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642824 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642995 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642860 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.642995 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.642969 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.644752 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.644730 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.644925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.644906 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.699772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.699696 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljgf\" (UniqueName: \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\") pod \"test-model-1410ed-kserve-6566755db9-6mgp4\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:55.721626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:55.721593 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:24:56.100287 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:56.100255 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:24:56.105797 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:24:56.105769 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e1efbad_84bd_4162_bc10_8097be701378.slice/crio-6ed0a487f1375ffd59b61aacaccd587153a325771c04f1b70c5191ac49caa671 WatchSource:0}: Error finding container 6ed0a487f1375ffd59b61aacaccd587153a325771c04f1b70c5191ac49caa671: Status 404 returned error can't find the container with id 6ed0a487f1375ffd59b61aacaccd587153a325771c04f1b70c5191ac49caa671 Nov 03 19:24:56.470692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:56.470614 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" event={"ID":"1e1efbad-84bd-4162-bc10-8097be701378","Type":"ContainerStarted","Data":"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38"} Nov 03 19:24:56.470692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:56.470660 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" event={"ID":"1e1efbad-84bd-4162-bc10-8097be701378","Type":"ContainerStarted","Data":"6ed0a487f1375ffd59b61aacaccd587153a325771c04f1b70c5191ac49caa671"} Nov 03 19:24:56.580024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:56.579972 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" podStartSLOduration=1.579959744 podStartE2EDuration="1.579959744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:56.579869028 +0000 UTC m=+2377.990969172" watchObservedRunningTime="2026-11-03 19:24:56.579959744 +0000 UTC m=+2377.991059868" Nov 03 19:24:56.721780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:24:56.721702 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:25:01.937492 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:25:01.937454 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:25:05.722776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:05.722742 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:25:05.735048 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:05.735024 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:25:06.553395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:06.553367 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:25:11.382498 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.382453 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:25:11.386585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.386564 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.425993 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bff69\" (UniqueName: \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.426150 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.426187 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.426215 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426426 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.426325 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.426501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.426435 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.432413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.432381 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:25:11.473799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.473766 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-332dac\"/\"default-dockercfg-kqvbd\"" Nov 03 19:25:11.473799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.473782 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-332dac\"/\"test-model-332dac-kserve-self-signed-certs\"" Nov 03 19:25:11.473799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.473797 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-332dac\"/\"kube-root-ca.crt\"" Nov 03 19:25:11.474069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.473782 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-332dac\"/\"openshift-service-ca.crt\"" Nov 03 19:25:11.527953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.527917 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.528178 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.527972 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.528178 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.528095 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.528423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.528273 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bff69\" (UniqueName: \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.528523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.528486 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.528659 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.528530 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.532648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.532618 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.534952 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.534928 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.535180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.535161 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.541548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.541509 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.552162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.544690 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.586373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.586338 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bff69\" (UniqueName: \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\") pod \"test-model-332dac-kserve-66c46464c-gw56d\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:11.700052 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:11.699966 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:12.083782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:12.083751 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:25:12.089472 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:25:12.089444 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23b61249_51e2_470f_8f79_ae622dd3060a.slice/crio-ad7fd59b33a8fd53f75dc36f5474cf34d0281c2f90682fc584596be685a57c7a WatchSource:0}: Error finding container ad7fd59b33a8fd53f75dc36f5474cf34d0281c2f90682fc584596be685a57c7a: Status 404 returned error can't find the container with id ad7fd59b33a8fd53f75dc36f5474cf34d0281c2f90682fc584596be685a57c7a Nov 03 19:25:12.591329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:12.591294 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" event={"ID":"23b61249-51e2-470f-8f79-ae622dd3060a","Type":"ContainerStarted","Data":"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f"} Nov 03 19:25:12.591720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:12.591335 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" event={"ID":"23b61249-51e2-470f-8f79-ae622dd3060a","Type":"ContainerStarted","Data":"ad7fd59b33a8fd53f75dc36f5474cf34d0281c2f90682fc584596be685a57c7a"} Nov 03 19:25:12.700208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:12.700176 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:12.702312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:12.702269 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" podStartSLOduration=1.702257919 podStartE2EDuration="1.702257919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:12.701962317 +0000 UTC m=+2394.113062452" watchObservedRunningTime="2026-11-03 19:25:12.702257919 +0000 UTC m=+2394.113358042" Nov 03 19:25:21.700358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:21.700324 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:21.713190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:21.713169 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:22.668510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:22.668480 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:25:31.937532 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:25:31.937497 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:25:35.379738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:35.379694 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:25:39.538843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.538810 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 19:25:39.539327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.539064 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" podUID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" containerName="maas-api" containerID="cri-o://277d495084dbe277aad4b763fe85adb85042e03961af24d46880715b033b7ca3" gracePeriod=30 Nov 03 19:25:39.782672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.782642 5209 generic.go:356] "Generic (PLEG): container finished" podID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" containerID="277d495084dbe277aad4b763fe85adb85042e03961af24d46880715b033b7ca3" exitCode=0 Nov 03 19:25:39.782843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.782724 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" event={"ID":"4f7f986b-1ca9-41a2-8756-f9adde07ead0","Type":"ContainerDied","Data":"277d495084dbe277aad4b763fe85adb85042e03961af24d46880715b033b7ca3"} Nov 03 19:25:39.825750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.825669 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 19:25:39.827755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.827729 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:25:39.935511 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.935475 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:25:39.983547 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.983504 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") pod \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " Nov 03 19:25:39.983748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.983625 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\") pod \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " Nov 03 19:25:39.983748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.983662 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\" (UniqueName: \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\") pod \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " Nov 03 19:25:39.983748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.983743 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\") pod \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\" (UID: \"4f7f986b-1ca9-41a2-8756-f9adde07ead0\") " Nov 03 19:25:39.984263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.984030 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca" pod "4f7f986b-1ca9-41a2-8756-f9adde07ead0" (UID: "4f7f986b-1ca9-41a2-8756-f9adde07ead0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:39.984387 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.984323 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f986b-1ca9-41a2-8756-f9adde07ead0-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:39.986680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.986649 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr" pod "4f7f986b-1ca9-41a2-8756-f9adde07ead0" (UID: "4f7f986b-1ca9-41a2-8756-f9adde07ead0"). InnerVolumeSpecName "kube-api-access-g4kpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:39.987239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.987216 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls" pod "4f7f986b-1ca9-41a2-8756-f9adde07ead0" (UID: "4f7f986b-1ca9-41a2-8756-f9adde07ead0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:39.989220 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:39.989193 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs" pod "4f7f986b-1ca9-41a2-8756-f9adde07ead0" (UID: "4f7f986b-1ca9-41a2-8756-f9adde07ead0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:40.085856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.085481 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:40.085856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.085514 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7f986b-1ca9-41a2-8756-f9adde07ead0-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:40.085856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.085546 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4kpr\" (UniqueName: \"kubernetes.io/projected/4f7f986b-1ca9-41a2-8756-f9adde07ead0-kube-api-access-g4kpr\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:40.790184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.790147 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" event={"ID":"4f7f986b-1ca9-41a2-8756-f9adde07ead0","Type":"ContainerDied","Data":"0a2d1a6be7bef9046f610e4ecaf2d902c04864e2cb22022fd194db5f036ef046"} Nov 03 19:25:40.790184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.790174 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8" Nov 03 19:25:40.790649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.790219 5209 scope.go:122] "RemoveContainer" containerID="277d495084dbe277aad4b763fe85adb85042e03961af24d46880715b033b7ca3" Nov 03 19:25:40.944455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.944423 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 19:25:40.949671 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:40.949647 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-90a9bf-847978f4d6-dhgw8"] Nov 03 19:25:42.699427 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:42.699395 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" path="/var/lib/kubelet/pods/4f7f986b-1ca9-41a2-8756-f9adde07ead0/volumes" Nov 03 19:25:44.829007 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:44.828966 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" podUID="7357c41d-2486-4dee-90f8-a767aae4f52b" containerName="payload-processing" containerID="cri-o://8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09" gracePeriod=25 Nov 03 19:25:44.936536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:44.936487 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" podUID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" containerName="payload-pre-processing" containerID="cri-o://779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b" gracePeriod=25 Nov 03 19:25:45.100851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.100772 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:25:45.118226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.118002 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:25:45.144524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.144485 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\" (UniqueName: \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\") pod \"7357c41d-2486-4dee-90f8-a767aae4f52b\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " Nov 03 19:25:45.144833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.144799 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\") pod \"7357c41d-2486-4dee-90f8-a767aae4f52b\" (UID: \"7357c41d-2486-4dee-90f8-a767aae4f52b\") " Nov 03 19:25:45.145528 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.145495 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume" pod "7357c41d-2486-4dee-90f8-a767aae4f52b" (UID: "7357c41d-2486-4dee-90f8-a767aae4f52b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:45.145693 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.145665 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357c41d-2486-4dee-90f8-a767aae4f52b-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.147478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.147447 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927" pod "7357c41d-2486-4dee-90f8-a767aae4f52b" (UID: "7357c41d-2486-4dee-90f8-a767aae4f52b"). InnerVolumeSpecName "kube-api-access-v5927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:45.161786 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.161756 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:25:45.197841 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.197814 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:25:45.247099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.247065 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\") pod \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " Nov 03 19:25:45.247312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.247136 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\") pod \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\" (UID: \"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f\") " Nov 03 19:25:45.247528 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.247498 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume" pod "bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" (UID: "bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:45.247733 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.247716 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.247795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.247739 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5927\" (UniqueName: \"kubernetes.io/projected/7357c41d-2486-4dee-90f8-a767aae4f52b-kube-api-access-v5927\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.249296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.249269 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp" pod "bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" (UID: "bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f"). InnerVolumeSpecName "kube-api-access-4s9sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:45.349298 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.349261 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f-kube-api-access-4s9sp\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.823065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.823030 5209 generic.go:356] "Generic (PLEG): container finished" podID="7357c41d-2486-4dee-90f8-a767aae4f52b" containerID="8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09" exitCode=0 Nov 03 19:25:45.823260 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.823072 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" Nov 03 19:25:45.823260 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.823110 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" event={"ID":"7357c41d-2486-4dee-90f8-a767aae4f52b","Type":"ContainerDied","Data":"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09"} Nov 03 19:25:45.823260 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.823185 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p" event={"ID":"7357c41d-2486-4dee-90f8-a767aae4f52b","Type":"ContainerDied","Data":"70539b2004228dd21a886f4f8fc926f6c071b8433a11255aaed8ad9918044e3e"} Nov 03 19:25:45.823260 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.823214 5209 scope.go:122] "RemoveContainer" containerID="8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09" Nov 03 19:25:45.824556 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.824535 5209 generic.go:356] "Generic (PLEG): container finished" podID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" containerID="779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b" exitCode=0 Nov 03 19:25:45.824661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.824569 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" event={"ID":"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f","Type":"ContainerDied","Data":"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b"} Nov 03 19:25:45.824661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.824574 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" Nov 03 19:25:45.824661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.824601 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm" event={"ID":"bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f","Type":"ContainerDied","Data":"4d98c62669bc0c42b2cb6455f942214bca55122f1e7b5b138dbf7aa31366e3fe"} Nov 03 19:25:45.833809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.833626 5209 scope.go:122] "RemoveContainer" containerID="8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09" Nov 03 19:25:45.834098 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:25:45.833866 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09\": container with ID starting with 8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09 not found: ID does not exist" containerID="8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09" Nov 03 19:25:45.834098 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.833892 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09"} err="failed to get container status \"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09\": rpc error: code = NotFound desc = could not find container \"8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09\": container with ID starting with 8fae2930fb046c4ea4c61c35a90a3ad8d8e0ad650eba2083e8174a78df4d4c09 not found: ID does not exist" Nov 03 19:25:45.834098 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.833909 5209 scope.go:122] "RemoveContainer" containerID="779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b" Nov 03 19:25:45.855388 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.855365 5209 scope.go:122] "RemoveContainer" containerID="779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b" Nov 03 19:25:45.855672 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:25:45.855651 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b\": container with ID starting with 779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b not found: ID does not exist" containerID="779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b" Nov 03 19:25:45.855741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.855681 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b"} err="failed to get container status \"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b\": rpc error: code = NotFound desc = could not find container \"779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b\": container with ID starting with 779649943bd313285074cff10c017bdfe1ff988a6364fe4731ffa6e6284c6f5b not found: ID does not exist" Nov 03 19:25:45.978358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.978328 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:25:45.982231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:45.982206 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2cbe3-779d658ff6-lkn5p"] Nov 03 19:25:46.084188 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:46.084150 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:25:46.087927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:46.087902 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2cbe3-5fc999d9c9-v67qm"] Nov 03 19:25:46.699273 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:46.699242 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7357c41d-2486-4dee-90f8-a767aae4f52b" path="/var/lib/kubelet/pods/7357c41d-2486-4dee-90f8-a767aae4f52b/volumes" Nov 03 19:25:46.699679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:46.699666 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" path="/var/lib/kubelet/pods/bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f/volumes" Nov 03 19:25:53.362235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.362196 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42"] Nov 03 19:25:53.362981 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.362962 5209 cpu_manager.go:414] "containerMap: removing container" podUID="7357c41d-2486-4dee-90f8-a767aae4f52b" containerName="payload-processing" Nov 03 19:25:53.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.362982 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7357c41d-2486-4dee-90f8-a767aae4f52b" containerName="payload-processing" Nov 03 19:25:53.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.362993 5209 cpu_manager.go:414] "containerMap: removing container" podUID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" containerName="payload-pre-processing" Nov 03 19:25:53.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.362998 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" containerName="payload-pre-processing" Nov 03 19:25:53.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.363024 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" containerName="maas-api" Nov 03 19:25:53.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.363030 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" containerName="maas-api" Nov 03 19:25:53.363249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.363128 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="7357c41d-2486-4dee-90f8-a767aae4f52b" containerName="payload-processing" Nov 03 19:25:53.363249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.363141 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2a92d9-5f8d-4f99-be10-cd54a4ee2f0f" containerName="payload-pre-processing" Nov 03 19:25:53.363249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.363153 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7f986b-1ca9-41a2-8756-f9adde07ead0" containerName="maas-api" Nov 03 19:25:53.367860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.367841 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.408778 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.408750 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:25:53.412640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.412622 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.421361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.421340 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-26cvs\"" Nov 03 19:25:53.457042 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457014 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457062 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457163 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457296 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457214 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457326 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthqs\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-kube-api-access-wthqs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457378 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457505 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457628 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457612 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.457676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.457662 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.463791 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.463767 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42"] Nov 03 19:25:53.472464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.472440 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:25:53.559239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559203 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.559423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559254 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559317 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559355 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559400 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559443 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthqs\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-kube-api-access-wthqs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559484 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559616 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrnn\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.559808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559674 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.559808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559706 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.559808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559711 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.559808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559784 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559834 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559839 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559904 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559943 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.559972 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.560038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560002 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560114 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.560317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560178 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560200 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.560317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560241 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.560456 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.560422 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.561865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.561844 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.562258 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.562239 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.617323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.617251 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.617323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.617310 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthqs\" (UniqueName: \"kubernetes.io/projected/b5f6b782-1721-4ad3-bf95-63ee93f71a7c-kube-api-access-wthqs\") pod \"maas-default-gateway-openshift-default-77f556d647-wxx42\" (UID: \"b5f6b782-1721-4ad3-bf95-63ee93f71a7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.661153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661106 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661185 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661231 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661296 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661405 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrnn\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661437 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661494 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661468 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661515 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661607 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661658 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.661802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661789 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.662025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.661984 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.662172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.662007 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.662414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.662391 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.663500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.663472 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.663747 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.663731 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.681161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.681143 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:53.728141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.725236 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrnn\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.728141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.725687 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kmlnh\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:53.728141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:53.725904 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:54.129687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.129658 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42"] Nov 03 19:25:54.171655 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:25:54.171623 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f6b782_1721_4ad3_bf95_63ee93f71a7c.slice/crio-72ae4b9720ac817057aa791793056b8cbe31de4889d59180bae447566e5aa579 WatchSource:0}: Error finding container 72ae4b9720ac817057aa791793056b8cbe31de4889d59180bae447566e5aa579: Status 404 returned error can't find the container with id 72ae4b9720ac817057aa791793056b8cbe31de4889d59180bae447566e5aa579 Nov 03 19:25:54.174030 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.173996 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.174156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.174067 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.174156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.174094 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.183585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.183524 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:25:54.188097 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:25:54.188064 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b3c85b_38db_4bcf_8ce3_2a55989e8c04.slice/crio-fc357adf6197ce86fd79be10397836ca931e744c97958d16ac8c38e564bbe9c6 WatchSource:0}: Error finding container fc357adf6197ce86fd79be10397836ca931e744c97958d16ac8c38e564bbe9c6: Status 404 returned error can't find the container with id fc357adf6197ce86fd79be10397836ca931e744c97958d16ac8c38e564bbe9c6 Nov 03 19:25:54.190563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.190534 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.190641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.190592 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.190641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.190619 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:25:54.934162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.934075 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" event={"ID":"b5f6b782-1721-4ad3-bf95-63ee93f71a7c","Type":"ContainerStarted","Data":"8ec2b44a71024ad8ad45f1f065b6c25fef1f1ec4ffb4c2621fd07ce1a807d842"} Nov 03 19:25:54.934607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.934173 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" event={"ID":"b5f6b782-1721-4ad3-bf95-63ee93f71a7c","Type":"ContainerStarted","Data":"72ae4b9720ac817057aa791793056b8cbe31de4889d59180bae447566e5aa579"} Nov 03 19:25:54.935887 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.935856 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" event={"ID":"22b3c85b-38db-4bcf-8ce3-2a55989e8c04","Type":"ContainerStarted","Data":"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667"} Nov 03 19:25:54.936006 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:54.935909 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" event={"ID":"22b3c85b-38db-4bcf-8ce3-2a55989e8c04","Type":"ContainerStarted","Data":"fc357adf6197ce86fd79be10397836ca931e744c97958d16ac8c38e564bbe9c6"} Nov 03 19:25:55.156892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.156834 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podStartSLOduration=2.15681898 podStartE2EDuration="2.15681898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:55.109679533 +0000 UTC m=+2436.520779681" watchObservedRunningTime="2026-11-03 19:25:55.15681898 +0000 UTC m=+2436.567919080" Nov 03 19:25:55.684150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.683469 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:25:55.694154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.691561 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:25:55.694154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.691709 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:25:55.730186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.726259 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:25:55.730186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.727918 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:25:55.730186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:55.727984 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:25:56.688465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:56.688431 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:25:56.689031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:56.688998 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:25:56.736172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:56.728515 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:25:56.736172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:56.728576 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:25:57.685510 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:57.685472 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:25:57.685810 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:57.685783 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:25:57.726577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:57.726540 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:25:57.727194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:57.727161 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:25:58.684492 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:58.684455 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:25:58.684785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:58.684752 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:25:58.727500 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:58.727465 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:25:58.728067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:58.728038 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:25:59.689142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:59.686464 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:25:59.689142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:59.686520 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:25:59.730869 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:59.730833 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:25:59.731519 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:25:59.731481 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:26:00.147561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.147525 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" containerID="cri-o://f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8" gracePeriod=45 Nov 03 19:26:00.185836 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.185301 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" containerID="cri-o://6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb" gracePeriod=45 Nov 03 19:26:00.517164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.516052 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:26:00.518563 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.518544 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:26:00.592231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592187 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.592387 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592240 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\" (UniqueName: \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592387 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592296 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\" (UniqueName: \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592394 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\" (UniqueName: \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.592493 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592434 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592498 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592531 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592611 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592563 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\") pod \"1922c791-3b1f-4594-84cc-1bb90fb16d52\" (UID: \"1922c791-3b1f-4594-84cc-1bb90fb16d52\") " Nov 03 19:26:00.592736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592620 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.592736 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592663 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.592837 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592807 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.592913 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.592880 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\") pod \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\" (UID: \"5e8707c1-0c6f-40ad-87ae-39a0b503ca48\") " Nov 03 19:26:00.593212 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.593187 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.593859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.593660 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.593859 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.593695 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.594037 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.593873 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.594775 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.594725 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.596067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.596039 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:00.596186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.596131 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.596186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.596111 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:00.598363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.598340 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.598558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.598346 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd" pod "1922c791-3b1f-4594-84cc-1bb90fb16d52" (UID: "1922c791-3b1f-4594-84cc-1bb90fb16d52"). InnerVolumeSpecName "kube-api-access-mrdqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:00.598677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.598520 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:00.600270 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.600244 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr" pod "5e8707c1-0c6f-40ad-87ae-39a0b503ca48" (UID: "5e8707c1-0c6f-40ad-87ae-39a0b503ca48"). InnerVolumeSpecName "kube-api-access-8vffr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:00.604447 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604428 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604451 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922c791-3b1f-4594-84cc-1bb90fb16d52-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604467 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604480 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604492 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604504 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.604543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.604516 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.629704 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.629651 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podStartSLOduration=7.629632831 podStartE2EDuration="7.629632831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:55.238480002 +0000 UTC m=+2436.649580138" watchObservedRunningTime="2026-11-03 19:26:00.629632831 +0000 UTC m=+2442.040732959" Nov 03 19:26:00.683512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.683476 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:26:00.683696 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.683551 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:26:00.705597 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.705565 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.705820 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.705802 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdqd\" (UniqueName: \"kubernetes.io/projected/1922c791-3b1f-4594-84cc-1bb90fb16d52-kube-api-access-mrdqd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.705929 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.705916 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vffr\" (UniqueName: \"kubernetes.io/projected/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-kube-api-access-8vffr\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.706030 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.706017 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922c791-3b1f-4594-84cc-1bb90fb16d52-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.706307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.706291 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8707c1-0c6f-40ad-87ae-39a0b503ca48-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:00.726802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.726737 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:26:00.727004 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.726974 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:26:00.986613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.986516 5209 generic.go:356] "Generic (PLEG): container finished" podID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerID="6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb" exitCode=0 Nov 03 19:26:00.986613 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.986562 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" Nov 03 19:26:00.987185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.986612 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" event={"ID":"5e8707c1-0c6f-40ad-87ae-39a0b503ca48","Type":"ContainerDied","Data":"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb"} Nov 03 19:26:00.987185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.986659 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf" event={"ID":"5e8707c1-0c6f-40ad-87ae-39a0b503ca48","Type":"ContainerDied","Data":"dffe7e70df9f8e24a51aaa1add48d2eb4a8c7b775135d0d837cbddb8f06fba2b"} Nov 03 19:26:00.987185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.986681 5209 scope.go:122] "RemoveContainer" containerID="6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb" Nov 03 19:26:00.988331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.988305 5209 generic.go:356] "Generic (PLEG): container finished" podID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerID="f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8" exitCode=0 Nov 03 19:26:00.988445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.988352 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" event={"ID":"1922c791-3b1f-4594-84cc-1bb90fb16d52","Type":"ContainerDied","Data":"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8"} Nov 03 19:26:00.988445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.988364 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" Nov 03 19:26:00.988445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.988380 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" event={"ID":"1922c791-3b1f-4594-84cc-1bb90fb16d52","Type":"ContainerDied","Data":"65e1d274f2094597fad48d19218d183fc4d76bf241abb89af9395bced3580fff"} Nov 03 19:26:00.998441 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.998420 5209 scope.go:122] "RemoveContainer" containerID="6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb" Nov 03 19:26:00.998762 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:26:00.998732 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb\": container with ID starting with 6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb not found: ID does not exist" containerID="6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb" Nov 03 19:26:00.998850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.998775 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb"} err="failed to get container status \"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb\": rpc error: code = NotFound desc = could not find container \"6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb\": container with ID starting with 6994ea59d28748cf64ebc7b90ad5ead18489859ab4df551f106a228ede2f77cb not found: ID does not exist" Nov 03 19:26:00.998850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:00.998801 5209 scope.go:122] "RemoveContainer" containerID="f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8" Nov 03 19:26:01.011350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.011330 5209 scope.go:122] "RemoveContainer" containerID="f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8" Nov 03 19:26:01.011680 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:26:01.011651 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8\": container with ID starting with f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8 not found: ID does not exist" containerID="f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8" Nov 03 19:26:01.011822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.011691 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8"} err="failed to get container status \"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8\": rpc error: code = NotFound desc = could not find container \"f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8\": container with ID starting with f354144a7694ed132c7ef982010a37d08b60c516de448b99595ffced614fc6e8 not found: ID does not exist" Nov 03 19:26:01.142113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.142078 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:26:01.146278 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.146247 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh"] Nov 03 19:26:01.249032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.248946 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:26:01.255476 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.255445 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/premium-w0-92596a-kserve-548bdd4bc8-4s8mf"] Nov 03 19:26:01.457620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.457580 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-2-w0-92596a-kserve-7f4f8c7546-2f6dh" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:26:01.682190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.682153 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:26:01.682406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.682217 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 19:26:01.727366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.727334 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:26:01.727587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:01.727382 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 19:26:01.940105 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:26:01.939997 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:26:02.707153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.706347 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" path="/var/lib/kubelet/pods/1922c791-3b1f-4594-84cc-1bb90fb16d52/volumes" Nov 03 19:26:02.707153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.707018 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" path="/var/lib/kubelet/pods/5e8707c1-0c6f-40ad-87ae-39a0b503ca48/volumes" Nov 03 19:26:02.861674 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.861638 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:26:02.862190 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.862163 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:26:02.863522 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.863502 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" Nov 03 19:26:02.912166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.911927 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:26:02.912404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.912368 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:26:02.913250 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:02.913230 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:26:08.482015 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.481978 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:26:08.482640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482602 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" Nov 03 19:26:08.482640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482617 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" Nov 03 19:26:08.482640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482629 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" Nov 03 19:26:08.482640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482634 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" Nov 03 19:26:08.482803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482724 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8707c1-0c6f-40ad-87ae-39a0b503ca48" containerName="main" Nov 03 19:26:08.482803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.482732 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1922c791-3b1f-4594-84cc-1bb90fb16d52" containerName="main" Nov 03 19:26:08.489342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.489314 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.585189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.585155 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:26:08.602715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.602666 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.602865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.602734 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.602865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.602794 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.602865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.602815 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.603021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.602995 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.603130 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.603104 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.603200 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.603186 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdqv\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.603265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.603252 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.603304 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.603279 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704670 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704620 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704670 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704678 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704709 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704731 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704844 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.704933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704926 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.704976 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdqv\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705063 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705094 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705152 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705189 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705266 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705386 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.705556 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.705512 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.707266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.707244 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.707541 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.707527 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.762152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.762079 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.762255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.762174 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdqv\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\") pod \"maas-default-gateway-openshift-default-77f556d647-2m425\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:08.806852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:08.806828 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:09.232454 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:09.232402 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:26:09.237549 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:26:09.237517 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68800080_3f84_4b48_a0ac_31ab9d9b5119.slice/crio-6ec4c9cdefac8d07135bf02fa8a9690665f8ad2e4ed72d7483a4c6b17fff4948 WatchSource:0}: Error finding container 6ec4c9cdefac8d07135bf02fa8a9690665f8ad2e4ed72d7483a4c6b17fff4948: Status 404 returned error can't find the container with id 6ec4c9cdefac8d07135bf02fa8a9690665f8ad2e4ed72d7483a4c6b17fff4948 Nov 03 19:26:09.239959 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:09.239924 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:26:09.240056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:09.239993 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:26:09.240056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:09.240023 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:26:10.053097 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.053065 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" event={"ID":"68800080-3f84-4b48-a0ac-31ab9d9b5119","Type":"ContainerStarted","Data":"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af"} Nov 03 19:26:10.053559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.053145 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" event={"ID":"68800080-3f84-4b48-a0ac-31ab9d9b5119","Type":"ContainerStarted","Data":"6ec4c9cdefac8d07135bf02fa8a9690665f8ad2e4ed72d7483a4c6b17fff4948"} Nov 03 19:26:10.167547 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.167498 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podStartSLOduration=2.167483424 podStartE2EDuration="2.167483424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:10.16519612 +0000 UTC m=+2451.576296263" watchObservedRunningTime="2026-11-03 19:26:10.167483424 +0000 UTC m=+2451.578583547" Nov 03 19:26:10.807414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.807373 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:10.809143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.809100 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:10.809274 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:10.809166 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:11.807586 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:11.807554 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:11.807996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:11.807606 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:12.807885 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:12.807527 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:12.807885 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:12.807587 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:13.811557 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:13.811515 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:13.812193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:13.812158 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:14.808143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:14.807554 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:14.808143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:14.807638 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:15.807558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:15.807525 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:15.807934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:15.807583 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:16.807909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:16.807865 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2m425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:26:16.808314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:16.807936 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 19:26:17.956838 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:17.956796 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:17.957356 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:17.957335 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:17.958176 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:17.958154 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:26:31.937648 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:26:31.937607 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:26:32.835979 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:26:32.835939 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:27:01.936721 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:27:01.936681 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:27:07.159846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:07.159811 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:27:07.160414 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:07.160026 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" containerID="cri-o://0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f" gracePeriod=30 Nov 03 19:27:12.413074 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.413042 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:27:12.449238 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449183 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449238 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449232 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449384 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449297 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449402 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449445 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449452 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:12.449504 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449500 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449665 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449576 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449633 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:12.449720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449686 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449821 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449739 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\") pod \"7e09a625-6a77-44a6-952a-41846b97e38d\" (UID: \"7e09a625-6a77-44a6-952a-41846b97e38d\") " Nov 03 19:27:12.449821 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449795 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:12.449941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449863 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:12.449941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.449927 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:12.450278 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.450252 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.450410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.450280 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.450410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.450295 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.450410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.450309 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e09a625-6a77-44a6-952a-41846b97e38d-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.450410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.450322 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.451565 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.451536 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:12.451565 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.451551 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:12.451699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.451576 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:27:12.451895 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.451877 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl" pod "7e09a625-6a77-44a6-952a-41846b97e38d" (UID: "7e09a625-6a77-44a6-952a-41846b97e38d"). InnerVolumeSpecName "kube-api-access-8lssl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:12.551201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.551174 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e09a625-6a77-44a6-952a-41846b97e38d-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.551201 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.551197 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e09a625-6a77-44a6-952a-41846b97e38d-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.551379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.551208 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lssl\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-kube-api-access-8lssl\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.551379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.551217 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e09a625-6a77-44a6-952a-41846b97e38d-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:12.556938 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.556911 5209 generic.go:356] "Generic (PLEG): container finished" podID="7e09a625-6a77-44a6-952a-41846b97e38d" containerID="0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f" exitCode=0 Nov 03 19:27:12.557062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.556958 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" Nov 03 19:27:12.557062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.556995 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" event={"ID":"7e09a625-6a77-44a6-952a-41846b97e38d","Type":"ContainerDied","Data":"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f"} Nov 03 19:27:12.557062 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.557048 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6" event={"ID":"7e09a625-6a77-44a6-952a-41846b97e38d","Type":"ContainerDied","Data":"77aa63e73f69c3dfda8b5e35b780807fda40b4c9574f4b4ba1ad5796da002ce5"} Nov 03 19:27:12.557209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.557069 5209 scope.go:122] "RemoveContainer" containerID="0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f" Nov 03 19:27:12.568203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.568185 5209 scope.go:122] "RemoveContainer" containerID="0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f" Nov 03 19:27:12.568480 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:27:12.568457 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f\": container with ID starting with 0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f not found: ID does not exist" containerID="0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f" Nov 03 19:27:12.568579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.568487 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f"} err="failed to get container status \"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f\": rpc error: code = NotFound desc = could not find container \"0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f\": container with ID starting with 0af79a7224d22639c04141871f309022e5b516665578f3b80a2f2fcba727f72f not found: ID does not exist" Nov 03 19:27:12.678967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.678933 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:27:12.683128 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.683091 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4kzt6"] Nov 03 19:27:12.699363 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:12.699305 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" path="/var/lib/kubelet/pods/7e09a625-6a77-44a6-952a-41846b97e38d/volumes" Nov 03 19:27:31.939270 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:27:31.939185 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:27:32.217707 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:32.217620 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:27:34.991040 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:34.991000 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:27:34.991577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:34.991227 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" containerID="cri-o://fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512" gracePeriod=30 Nov 03 19:27:37.108502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:37.108468 5209 patch_prober.go:29] interesting pod/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:27:37.108901 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:37.108529 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:27:40.250111 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.250086 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:27:40.285976 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.285910 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.285976 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.285961 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286014 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286063 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286106 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286246 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286330 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286349 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:40.286523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286357 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:40.286523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286381 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286466 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\") pod \"8f712aea-e9d3-4924-88d7-11bff330aabc\" (UID: \"8f712aea-e9d3-4924-88d7-11bff330aabc\") " Nov 03 19:27:40.286714 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286553 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:40.286888 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286862 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:40.286960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286888 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.286960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286908 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.286960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286789 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:40.286960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.286923 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.288407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.288379 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:40.288502 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.288427 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "kube-api-access-8dpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:40.288604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.288586 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:40.288660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.288595 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo" pod "8f712aea-e9d3-4924-88d7-11bff330aabc" (UID: "8f712aea-e9d3-4924-88d7-11bff330aabc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:27:40.388641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388608 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.388641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388636 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.388641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388645 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.388890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388656 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.388890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388665 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpn9\" (UniqueName: \"kubernetes.io/projected/8f712aea-e9d3-4924-88d7-11bff330aabc-kube-api-access-8dpn9\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.388890 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.388673 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f712aea-e9d3-4924-88d7-11bff330aabc-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:40.751635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.751604 5209 generic.go:356] "Generic (PLEG): container finished" podID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerID="fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512" exitCode=0 Nov 03 19:27:40.751782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.751675 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" Nov 03 19:27:40.751782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.751689 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerDied","Data":"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512"} Nov 03 19:27:40.751782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.751749 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t" event={"ID":"8f712aea-e9d3-4924-88d7-11bff330aabc","Type":"ContainerDied","Data":"7c9ef713467835d28d0ee9975a378e007b56ed9e049598f7d5f7a22ad21357d9"} Nov 03 19:27:40.751782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.751780 5209 scope.go:122] "RemoveContainer" containerID="fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512" Nov 03 19:27:40.761359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.761341 5209 scope.go:122] "RemoveContainer" containerID="99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8" Nov 03 19:27:40.769842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.769821 5209 scope.go:122] "RemoveContainer" containerID="fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512" Nov 03 19:27:40.770093 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:27:40.770074 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512\": container with ID starting with fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512 not found: ID does not exist" containerID="fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512" Nov 03 19:27:40.770209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.770102 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512"} err="failed to get container status \"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512\": rpc error: code = NotFound desc = could not find container \"fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512\": container with ID starting with fba16fe4ffb99fca278a6c26c0fb12677bdd95e0d7e40c359ab47e0606754512 not found: ID does not exist" Nov 03 19:27:40.770209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.770137 5209 scope.go:122] "RemoveContainer" containerID="99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8" Nov 03 19:27:40.770383 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:27:40.770368 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8\": container with ID starting with 99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8 not found: ID does not exist" containerID="99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8" Nov 03 19:27:40.770432 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.770389 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8"} err="failed to get container status \"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8\": rpc error: code = NotFound desc = could not find container \"99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8\": container with ID starting with 99ae5a40633c8933cab7a9859d2a67b3db1acf7d8b3e1618f4cecc9f70ccd2e8 not found: ID does not exist" Nov 03 19:27:40.909413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.909383 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:27:40.955969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:40.955935 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d2cbe3-openshift-default-d68b74ff5-5qc5t"] Nov 03 19:27:42.699088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:42.699056 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" path="/var/lib/kubelet/pods/8f712aea-e9d3-4924-88d7-11bff330aabc/volumes" Nov 03 19:27:55.732332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:55.732284 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 19:27:55.732983 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:55.732537 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" podUID="fdf5bb02-37fa-48aa-9988-8e55840f1312" containerName="maas-api" containerID="cri-o://0cd0a417f711851d7fe3f4c3400fa77e148b2d7d2bca65ed50b84eeffcc0189a" gracePeriod=30 Nov 03 19:27:55.858422 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:55.858393 5209 generic.go:356] "Generic (PLEG): container finished" podID="fdf5bb02-37fa-48aa-9988-8e55840f1312" containerID="0cd0a417f711851d7fe3f4c3400fa77e148b2d7d2bca65ed50b84eeffcc0189a" exitCode=0 Nov 03 19:27:55.858580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:55.858480 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" event={"ID":"fdf5bb02-37fa-48aa-9988-8e55840f1312","Type":"ContainerDied","Data":"0cd0a417f711851d7fe3f4c3400fa77e148b2d7d2bca65ed50b84eeffcc0189a"} Nov 03 19:27:56.025096 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.025068 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 19:27:56.183719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.183682 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\") pod \"fdf5bb02-37fa-48aa-9988-8e55840f1312\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " Nov 03 19:27:56.183927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.183759 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\") pod \"fdf5bb02-37fa-48aa-9988-8e55840f1312\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " Nov 03 19:27:56.184192 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.184160 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca" pod "fdf5bb02-37fa-48aa-9988-8e55840f1312" (UID: "fdf5bb02-37fa-48aa-9988-8e55840f1312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:56.184356 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.184330 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\") pod \"fdf5bb02-37fa-48aa-9988-8e55840f1312\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " Nov 03 19:27:56.184495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.184424 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\" (UniqueName: \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\") pod \"fdf5bb02-37fa-48aa-9988-8e55840f1312\" (UID: \"fdf5bb02-37fa-48aa-9988-8e55840f1312\") " Nov 03 19:27:56.184992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.184965 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf5bb02-37fa-48aa-9988-8e55840f1312-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:56.186643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.186615 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs" pod "fdf5bb02-37fa-48aa-9988-8e55840f1312" (UID: "fdf5bb02-37fa-48aa-9988-8e55840f1312"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:56.186732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.186718 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8" pod "fdf5bb02-37fa-48aa-9988-8e55840f1312" (UID: "fdf5bb02-37fa-48aa-9988-8e55840f1312"). InnerVolumeSpecName "kube-api-access-5sdb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:56.187036 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.187006 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls" pod "fdf5bb02-37fa-48aa-9988-8e55840f1312" (UID: "fdf5bb02-37fa-48aa-9988-8e55840f1312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:56.286446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.286358 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:56.286446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.286397 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf5bb02-37fa-48aa-9988-8e55840f1312-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:56.286446 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.286414 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdb8\" (UniqueName: \"kubernetes.io/projected/fdf5bb02-37fa-48aa-9988-8e55840f1312-kube-api-access-5sdb8\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:56.866104 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.866075 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" Nov 03 19:27:56.866598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.866070 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs" event={"ID":"fdf5bb02-37fa-48aa-9988-8e55840f1312","Type":"ContainerDied","Data":"ed0f77f58e1389ded3f67501407868a15d7d9c13bfea6871e61401f89bbfa478"} Nov 03 19:27:56.866598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:56.866201 5209 scope.go:122] "RemoveContainer" containerID="0cd0a417f711851d7fe3f4c3400fa77e148b2d7d2bca65ed50b84eeffcc0189a" Nov 03 19:27:57.021366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:57.021285 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 19:27:57.067906 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:57.067868 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-92596a-56dddbf54d-kdxvs"] Nov 03 19:27:58.699415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:27:58.699384 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf5bb02-37fa-48aa-9988-8e55840f1312" path="/var/lib/kubelet/pods/fdf5bb02-37fa-48aa-9988-8e55840f1312/volumes" Nov 03 19:28:01.937884 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:28:01.937844 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:28:04.560262 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:04.560229 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:28:04.560710 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:04.560454 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" containerID="cri-o://8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480" gracePeriod=30 Nov 03 19:28:09.821018 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.820992 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:28:09.841258 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841205 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841258 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841240 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841259 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841309 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841361 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841445 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841398 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841475 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841472 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:09.841672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841495 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:09.841672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841529 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841623 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\") pod \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\" (UID: \"eb44d65f-6697-4923-9fb5-7d5f8a89afb8\") " Nov 03 19:28:09.841943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841729 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:09.841943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.841847 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:28:09.842053 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842004 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:09.842246 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842227 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.842323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842251 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.842323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842267 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.842323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842280 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.842323 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.842294 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.843496 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.843470 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:28:09.843605 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.843533 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:09.843920 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.843901 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "kube-api-access-8cgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:09.843998 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.843911 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy" pod "eb44d65f-6697-4923-9fb5-7d5f8a89afb8" (UID: "eb44d65f-6697-4923-9fb5-7d5f8a89afb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:09.943371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.943338 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.943371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.943366 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.943371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.943377 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.943596 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.943387 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgzn\" (UniqueName: \"kubernetes.io/projected/eb44d65f-6697-4923-9fb5-7d5f8a89afb8-kube-api-access-8cgzn\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:09.956972 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.956937 5209 generic.go:356] "Generic (PLEG): container finished" podID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerID="8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480" exitCode=0 Nov 03 19:28:09.957150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.956982 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" Nov 03 19:28:09.957150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.957017 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" event={"ID":"eb44d65f-6697-4923-9fb5-7d5f8a89afb8","Type":"ContainerDied","Data":"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480"} Nov 03 19:28:09.957150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.957075 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8" event={"ID":"eb44d65f-6697-4923-9fb5-7d5f8a89afb8","Type":"ContainerDied","Data":"265bb57a20c7709ea24fcdf2737bf57c7fc5ff6f7603f404a58e89b14049480c"} Nov 03 19:28:09.957150 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.957108 5209 scope.go:122] "RemoveContainer" containerID="8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480" Nov 03 19:28:09.969477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.969459 5209 scope.go:122] "RemoveContainer" containerID="8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480" Nov 03 19:28:09.969744 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:28:09.969726 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480\": container with ID starting with 8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480 not found: ID does not exist" containerID="8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480" Nov 03 19:28:09.969843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:09.969752 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480"} err="failed to get container status \"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480\": rpc error: code = NotFound desc = could not find container \"8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480\": container with ID starting with 8193eff6fc7fb1160cf6a67ba895293e933e83884634e955cac2264414c4f480 not found: ID does not exist" Nov 03 19:28:10.088091 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:10.088052 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:28:10.092236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:10.092173 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-4zjz8"] Nov 03 19:28:10.698954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:10.698851 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" path="/var/lib/kubelet/pods/eb44d65f-6697-4923-9fb5-7d5f8a89afb8/volumes" Nov 03 19:28:17.028023 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:17.027990 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:28:31.937408 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:28:31.937366 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:28:38.597600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:38.597559 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:28:53.617420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.617377 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" podUID="1e1efbad-84bd-4162-bc10-8097be701378" containerName="main" containerID="cri-o://bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38" gracePeriod=45 Nov 03 19:28:53.885035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.884975 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:28:53.922372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922345 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922380 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922409 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922438 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922473 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\" (UniqueName: \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922617 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\") pod \"1e1efbad-84bd-4162-bc10-8097be701378\" (UID: \"1e1efbad-84bd-4162-bc10-8097be701378\") " Nov 03 19:28:53.922753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922643 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:53.922842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922743 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:53.922899 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.922877 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:53.923147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.923096 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:53.923245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.923148 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:53.923245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.923166 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:53.924717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.924690 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:28:53.924827 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.924740 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:53.924827 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:53.924780 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf" pod "1e1efbad-84bd-4162-bc10-8097be701378" (UID: "1e1efbad-84bd-4162-bc10-8097be701378"). InnerVolumeSpecName "kube-api-access-5ljgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:54.024637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.024600 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e1efbad-84bd-4162-bc10-8097be701378-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:54.024637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.024629 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e1efbad-84bd-4162-bc10-8097be701378-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:54.024637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.024639 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljgf\" (UniqueName: \"kubernetes.io/projected/1e1efbad-84bd-4162-bc10-8097be701378-kube-api-access-5ljgf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:54.239924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.239842 5209 generic.go:356] "Generic (PLEG): container finished" podID="1e1efbad-84bd-4162-bc10-8097be701378" containerID="bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38" exitCode=0 Nov 03 19:28:54.239924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.239894 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" Nov 03 19:28:54.240132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.239929 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" event={"ID":"1e1efbad-84bd-4162-bc10-8097be701378","Type":"ContainerDied","Data":"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38"} Nov 03 19:28:54.240132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.239981 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4" event={"ID":"1e1efbad-84bd-4162-bc10-8097be701378","Type":"ContainerDied","Data":"6ed0a487f1375ffd59b61aacaccd587153a325771c04f1b70c5191ac49caa671"} Nov 03 19:28:54.240132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.240001 5209 scope.go:122] "RemoveContainer" containerID="bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38" Nov 03 19:28:54.249423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.249401 5209 scope.go:122] "RemoveContainer" containerID="bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38" Nov 03 19:28:54.249690 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:28:54.249663 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38\": container with ID starting with bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38 not found: ID does not exist" containerID="bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38" Nov 03 19:28:54.249738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.249690 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38"} err="failed to get container status \"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38\": rpc error: code = NotFound desc = could not find container \"bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38\": container with ID starting with bf1aaf90c5e1f57f2b2499d0405f46f5962b524299e76e59e8d827525bd6ae38 not found: ID does not exist" Nov 03 19:28:54.394432 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.394406 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:28:54.398230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.398206 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1410ed/test-model-1410ed-kserve-6566755db9-6mgp4"] Nov 03 19:28:54.699695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:28:54.699660 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1efbad-84bd-4162-bc10-8097be701378" path="/var/lib/kubelet/pods/1e1efbad-84bd-4162-bc10-8097be701378/volumes" Nov 03 19:29:01.937418 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:29:01.937337 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:29:03.315484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:03.315450 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:29:15.215191 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:15.215153 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:29:15.320925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:15.320888 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:29:18.336908 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.336867 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" podUID="23b61249-51e2-470f-8f79-ae622dd3060a" containerName="main" containerID="cri-o://e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f" gracePeriod=45 Nov 03 19:29:18.599854 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.599795 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:29:18.709543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709509 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.709754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709550 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.709754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709599 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\" (UniqueName: \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.709754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709667 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\" (UniqueName: \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.709754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709696 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.709754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709740 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\") pod \"23b61249-51e2-470f-8f79-ae622dd3060a\" (UID: \"23b61249-51e2-470f-8f79-ae622dd3060a\") " Nov 03 19:29:18.710183 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.709910 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:18.710307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.710285 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:18.710374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.710300 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:18.710432 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.710372 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:18.712558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.712528 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:29:18.714209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.714177 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "kube-api-access-bff69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:18.714301 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.714225 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm" pod "23b61249-51e2-470f-8f79-ae622dd3060a" (UID: "23b61249-51e2-470f-8f79-ae622dd3060a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:18.810996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.810954 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:18.810996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.810976 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b61249-51e2-470f-8f79-ae622dd3060a-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:18.810996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.810986 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bff69\" (UniqueName: \"kubernetes.io/projected/23b61249-51e2-470f-8f79-ae622dd3060a-kube-api-access-bff69\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:18.810996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.810999 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:18.811317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:18.811008 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b61249-51e2-470f-8f79-ae622dd3060a-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:19.401179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.401138 5209 generic.go:356] "Generic (PLEG): container finished" podID="23b61249-51e2-470f-8f79-ae622dd3060a" containerID="e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f" exitCode=0 Nov 03 19:29:19.401179 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.401179 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" Nov 03 19:29:19.401703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.401217 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" event={"ID":"23b61249-51e2-470f-8f79-ae622dd3060a","Type":"ContainerDied","Data":"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f"} Nov 03 19:29:19.401703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.401268 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d" event={"ID":"23b61249-51e2-470f-8f79-ae622dd3060a","Type":"ContainerDied","Data":"ad7fd59b33a8fd53f75dc36f5474cf34d0281c2f90682fc584596be685a57c7a"} Nov 03 19:29:19.401703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.401294 5209 scope.go:122] "RemoveContainer" containerID="e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f" Nov 03 19:29:19.411949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.411934 5209 scope.go:122] "RemoveContainer" containerID="e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f" Nov 03 19:29:19.412210 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:29:19.412190 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f\": container with ID starting with e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f not found: ID does not exist" containerID="e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f" Nov 03 19:29:19.412265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.412218 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f"} err="failed to get container status \"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f\": rpc error: code = NotFound desc = could not find container \"e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f\": container with ID starting with e53396d11fb04ab5240108d749eabd90ec2d46bda40f76f80db56e457b44952f not found: ID does not exist" Nov 03 19:29:19.555942 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.555911 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:29:19.602350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:19.602320 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-332dac/test-model-332dac-kserve-66c46464c-gw56d"] Nov 03 19:29:20.321397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:20.321359 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" containerID="cri-o://0d60cfbbaf72d10d2c0aa720d9b655c7bf137d7993f2e9e9c0d915322464cf64" gracePeriod=25 Nov 03 19:29:20.698865 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:20.698826 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b61249-51e2-470f-8f79-ae622dd3060a" path="/var/lib/kubelet/pods/23b61249-51e2-470f-8f79-ae622dd3060a/volumes" Nov 03 19:29:21.407265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:21.407225 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:29:31.407626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:31.407586 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:29:31.936937 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:29:31.936901 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:29:39.193208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193174 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:29:39.193714 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193691 5209 cpu_manager.go:414] "containerMap: removing container" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:29:39.193714 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193704 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193727 5209 cpu_manager.go:414] "containerMap: removing container" podUID="fdf5bb02-37fa-48aa-9988-8e55840f1312" containerName="maas-api" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193733 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf5bb02-37fa-48aa-9988-8e55840f1312" containerName="maas-api" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193742 5209 cpu_manager.go:414] "containerMap: removing container" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193748 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193756 5209 cpu_manager.go:414] "containerMap: removing container" podUID="23b61249-51e2-470f-8f79-ae622dd3060a" containerName="main" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193760 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b61249-51e2-470f-8f79-ae622dd3060a" containerName="main" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193770 5209 cpu_manager.go:414] "containerMap: removing container" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193777 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193787 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1e1efbad-84bd-4162-bc10-8097be701378" containerName="main" Nov 03 19:29:39.193794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.193792 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1efbad-84bd-4162-bc10-8097be701378" containerName="main" Nov 03 19:29:39.194392 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194365 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:29:39.194539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194408 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf5bb02-37fa-48aa-9988-8e55840f1312" containerName="maas-api" Nov 03 19:29:39.194539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194429 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e09a625-6a77-44a6-952a-41846b97e38d" containerName="istio-proxy" Nov 03 19:29:39.194539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194439 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b61249-51e2-470f-8f79-ae622dd3060a" containerName="main" Nov 03 19:29:39.194539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194461 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb44d65f-6697-4923-9fb5-7d5f8a89afb8" containerName="istio-proxy" Nov 03 19:29:39.194539 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.194481 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1efbad-84bd-4162-bc10-8097be701378" containerName="main" Nov 03 19:29:39.206157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.206110 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.253676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.253642 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:29:39.296689 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.296660 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-73a216\"/\"openshift-service-ca.crt\"" Nov 03 19:29:39.296846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.296697 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-73a216\"/\"test-model-73a216-kserve-self-signed-certs\"" Nov 03 19:29:39.296846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.296773 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-73a216\"/\"kube-root-ca.crt\"" Nov 03 19:29:39.299449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.299435 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-73a216\"/\"default-dockercfg-xvgbb\"" Nov 03 19:29:39.344902 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.344863 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlq7\" (UniqueName: \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.345056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.344985 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.345056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.345028 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.345181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.345070 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.345280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.345259 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.345374 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.345351 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.446732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446649 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.446732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446696 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.446732 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446723 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446764 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446787 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447009 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.446878 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlq7\" (UniqueName: \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.447103 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.447115 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.447415 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.447392 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.449245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.449223 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.449517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.449495 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.505036 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.505004 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlq7\" (UniqueName: \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\") pod \"test-model-73a216-kserve-7795f655c5-nbr2l\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.518090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.518063 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:39.892688 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:39.892657 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:29:39.898190 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:29:39.898158 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf95000b_df6e_4a58_a114_f4e5ca7a5382.slice/crio-c2821f133b1d1e5371cca71ddd8e5139fb68b1f63c4c6299edd3b39ccc645967 WatchSource:0}: Error finding container c2821f133b1d1e5371cca71ddd8e5139fb68b1f63c4c6299edd3b39ccc645967: Status 404 returned error can't find the container with id c2821f133b1d1e5371cca71ddd8e5139fb68b1f63c4c6299edd3b39ccc645967 Nov 03 19:29:40.533181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.533148 5209 generic.go:356] "Generic (PLEG): container finished" podID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerID="0d60cfbbaf72d10d2c0aa720d9b655c7bf137d7993f2e9e9c0d915322464cf64" exitCode=0 Nov 03 19:29:40.533648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.533225 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" event={"ID":"3b8f51c2-3bfb-44ee-9e3e-75491321966b","Type":"ContainerDied","Data":"0d60cfbbaf72d10d2c0aa720d9b655c7bf137d7993f2e9e9c0d915322464cf64"} Nov 03 19:29:40.535347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.535316 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" event={"ID":"af95000b-df6e-4a58-a114-f4e5ca7a5382","Type":"ContainerStarted","Data":"bdac714a86035bef0dce4c8563c05d792cf7ded4a34a1b36430061bf06616325"} Nov 03 19:29:40.535497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.535359 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" event={"ID":"af95000b-df6e-4a58-a114-f4e5ca7a5382","Type":"ContainerStarted","Data":"c2821f133b1d1e5371cca71ddd8e5139fb68b1f63c4c6299edd3b39ccc645967"} Nov 03 19:29:40.572234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.572213 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:29:40.644635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.644536 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" podStartSLOduration=1.6445230309999999 podStartE2EDuration="1.644523031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:40.644331232 +0000 UTC m=+2662.055431355" watchObservedRunningTime="2026-11-03 19:29:40.644523031 +0000 UTC m=+2662.055623154" Nov 03 19:29:40.658203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.658173 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\" (UniqueName: \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\") pod \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " Nov 03 19:29:40.658371 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.658298 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\") pod \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\" (UID: \"3b8f51c2-3bfb-44ee-9e3e-75491321966b\") " Nov 03 19:29:40.658640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.658617 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume" pod "3b8f51c2-3bfb-44ee-9e3e-75491321966b" (UID: "3b8f51c2-3bfb-44ee-9e3e-75491321966b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:29:40.660233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.660210 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k" pod "3b8f51c2-3bfb-44ee-9e3e-75491321966b" (UID: "3b8f51c2-3bfb-44ee-9e3e-75491321966b"). InnerVolumeSpecName "kube-api-access-45r6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:40.759645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.759610 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8f51c2-3bfb-44ee-9e3e-75491321966b-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:40.759892 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:40.759866 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45r6k\" (UniqueName: \"kubernetes.io/projected/3b8f51c2-3bfb-44ee-9e3e-75491321966b-kube-api-access-45r6k\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:41.519079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.519040 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:41.543372 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.543345 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" Nov 03 19:29:41.543815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.543387 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2" event={"ID":"3b8f51c2-3bfb-44ee-9e3e-75491321966b","Type":"ContainerDied","Data":"9d7bb82111ad884b7aa21b7fcc6983df1b1587bc6ee010bf2bf8cb8c9971d7cb"} Nov 03 19:29:41.543815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.543441 5209 scope.go:122] "RemoveContainer" containerID="0d60cfbbaf72d10d2c0aa720d9b655c7bf137d7993f2e9e9c0d915322464cf64" Nov 03 19:29:41.697206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.697175 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:29:41.743757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:41.743723 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1410ed-5b58cf6499-bh4c2"] Nov 03 19:29:42.698361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:42.698326 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" path="/var/lib/kubelet/pods/3b8f51c2-3bfb-44ee-9e3e-75491321966b/volumes" Nov 03 19:29:49.518529 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:49.518493 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:49.531230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:49.531209 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:49.605637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:49.605606 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:29:55.621750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:29:55.621713 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:30:00.174088 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.174048 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:00.174857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.174838 5209 cpu_manager.go:414] "containerMap: removing container" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:30:00.174918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.174862 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:30:00.174962 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.174928 5209 cpu_manager.go:414] "containerMap: removing container" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" Nov 03 19:30:00.174962 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.174938 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" Nov 03 19:30:00.175089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.175075 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8f51c2-3bfb-44ee-9e3e-75491321966b" containerName="payload-pre-processing" Nov 03 19:30:00.175164 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.175096 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f712aea-e9d3-4924-88d7-11bff330aabc" containerName="istio-proxy" Nov 03 19:30:00.178883 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.178861 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.191753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.191728 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:00.201316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.201288 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:00.204166 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.204108 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.206513 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.206489 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.218056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.218034 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:00.221615 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.221595 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:00.221726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.221708 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.226846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.226828 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.272161 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.271896 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:00.281780 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.281753 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:00.283181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283114 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6bf\" (UniqueName: \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.283305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283279 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.283375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283330 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn88h\" (UniqueName: \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.283375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283358 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.283498 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283423 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.283498 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.283466 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5dk\" (UniqueName: \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.286110 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.286088 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:00.317172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.317143 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:00.324173 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.324151 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:30:00.343159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.343093 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:00.384411 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384377 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.384582 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384422 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.384582 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384460 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5dk\" (UniqueName: \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.384582 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384505 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjzh\" (UniqueName: \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.384801 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384779 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6bf\" (UniqueName: \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.384864 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384842 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.384909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.384894 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvx8f\" (UniqueName: \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.385032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385014 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.385094 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385079 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn88h\" (UniqueName: \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.385155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385109 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.385204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385114 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.385574 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385557 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.385652 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.385636 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.442357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.442280 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6bf\" (UniqueName: \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.442357 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.442331 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn88h\" (UniqueName: \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\") pod \"maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.442532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.442332 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5dk\" (UniqueName: \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.486312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486266 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.486312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486314 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjzh\" (UniqueName: \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.486569 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486385 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.486569 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486428 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx8f\" (UniqueName: \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.487005 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486979 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.487093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.486985 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.501016 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.500993 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:00.523651 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.523627 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:00.531759 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.531739 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:00.548222 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.548195 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx8f\" (UniqueName: \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\") pod \"maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.548404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.548388 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjzh\" (UniqueName: \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\") pod \"maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.839912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.839882 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:00.846984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.846962 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:00.879506 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.879474 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:00.885001 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:30:00.884792 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4638a05d_61eb_4a81_b4dc_f47fd4fb8087.slice/crio-14a9e82af532d855cc4d6d942159783666ce837252cbb0ec544e5e9a7f6d3faf WatchSource:0}: Error finding container 14a9e82af532d855cc4d6d942159783666ce837252cbb0ec544e5e9a7f6d3faf: Status 404 returned error can't find the container with id 14a9e82af532d855cc4d6d942159783666ce837252cbb0ec544e5e9a7f6d3faf Nov 03 19:30:00.934366 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.933624 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:00.951159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:00.944619 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:00.951159 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:00.946005 5209 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab29859_8aec_40ac_ad30_bd0e8e91ab6e.slice/crio-8b1e3c9dff6e9c453e757be6d69d47a8a2888a6d822411bdaf3a8ad72f990da4\": RecentStats: unable to find data in memory cache]" Nov 03 19:30:00.956089 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:00.956022 5209 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca0a289_a7ba_4276_b72a_b0572dbe7195.slice/crio-0b68e768e592caf17b5d33a0ff110a2c4857ede06f5283598b8e68bd17282d6d\": RecentStats: unable to find data in memory cache]" Nov 03 19:30:01.219060 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.219026 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:01.226647 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.226624 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:01.235144 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:30:01.235104 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b428bc_a365_4e4b_ac0f_55bd3ceb7ec9.slice/crio-b2a3c3485307056a07d7894178de144fad6eed1efcf10bab8254926000632bc1 WatchSource:0}: Error finding container b2a3c3485307056a07d7894178de144fad6eed1efcf10bab8254926000632bc1: Status 404 returned error can't find the container with id b2a3c3485307056a07d7894178de144fad6eed1efcf10bab8254926000632bc1 Nov 03 19:30:01.670713 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.670671 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerStarted","Data":"abf32bf433d3eb9b9248ae1612f23fb5078a2981bd48c6e4ae6a754b861b050d"} Nov 03 19:30:01.670912 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.670740 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerStarted","Data":"4c60a35413c3b7d136a3185ac4968bd346074a73104bac74ece7f5f2236b143e"} Nov 03 19:30:01.672660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.672624 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerStarted","Data":"ffaff2f1255cc4e923ba2c40e921c64e2ff03e193df846baa1a7b00f006940d9"} Nov 03 19:30:01.672822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.672685 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerStarted","Data":"8b1e3c9dff6e9c453e757be6d69d47a8a2888a6d822411bdaf3a8ad72f990da4"} Nov 03 19:30:01.674462 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.674434 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerStarted","Data":"94707f1ae7bc03e7a8d24b8bb1553da00e33ad6ef3510db91639bbe176d82b99"} Nov 03 19:30:01.674588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.674488 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerStarted","Data":"b2a3c3485307056a07d7894178de144fad6eed1efcf10bab8254926000632bc1"} Nov 03 19:30:01.679969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.676263 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerStarted","Data":"6b8d914d2e5619545096e323cc8cfa9e7226f95e8b62e1a66150d16a6557926b"} Nov 03 19:30:01.679969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.676308 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerStarted","Data":"14a9e82af532d855cc4d6d942159783666ce837252cbb0ec544e5e9a7f6d3faf"} Nov 03 19:30:01.679969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.678916 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerStarted","Data":"6a982c13464e8e00163e45c5b7065e0292fa1a6d066ae4e283e4d0a3743efbb9"} Nov 03 19:30:01.679969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.678965 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerStarted","Data":"0b68e768e592caf17b5d33a0ff110a2c4857ede06f5283598b8e68bd17282d6d"} Nov 03 19:30:01.782230 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.782057 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" podStartSLOduration=1.782039489 podStartE2EDuration="1.782039489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:01.779039479 +0000 UTC m=+2683.190139626" watchObservedRunningTime="2026-11-03 19:30:01.782039489 +0000 UTC m=+2683.193139614" Nov 03 19:30:01.891240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.891190 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" podStartSLOduration=1.891177494 podStartE2EDuration="1.891177494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:01.889914187 +0000 UTC m=+2683.301014310" watchObservedRunningTime="2026-11-03 19:30:01.891177494 +0000 UTC m=+2683.302277617" Nov 03 19:30:01.979238 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:01.979203 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:30:01.999239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:01.999187 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" podStartSLOduration=1.999170532 podStartE2EDuration="1.999170532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:01.99816315 +0000 UTC m=+2683.409263269" watchObservedRunningTime="2026-11-03 19:30:01.999170532 +0000 UTC m=+2683.410270657" Nov 03 19:30:02.112054 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.111999 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" podStartSLOduration=2.111986729 podStartE2EDuration="2.111986729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.10946224 +0000 UTC m=+2683.520562381" watchObservedRunningTime="2026-11-03 19:30:02.111986729 +0000 UTC m=+2683.523086852" Nov 03 19:30:02.219830 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.219778 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" podStartSLOduration=2.219764066 podStartE2EDuration="2.219764066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.217374429 +0000 UTC m=+2683.628474561" watchObservedRunningTime="2026-11-03 19:30:02.219764066 +0000 UTC m=+2683.630864189" Nov 03 19:30:02.686497 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.686403 5209 generic.go:356] "Generic (PLEG): container finished" podID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerID="94707f1ae7bc03e7a8d24b8bb1553da00e33ad6ef3510db91639bbe176d82b99" exitCode=60 Nov 03 19:30:02.686660 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.686492 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerDied","Data":"94707f1ae7bc03e7a8d24b8bb1553da00e33ad6ef3510db91639bbe176d82b99"} Nov 03 19:30:02.686768 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.686751 5209 scope.go:122] "RemoveContainer" containerID="94707f1ae7bc03e7a8d24b8bb1553da00e33ad6ef3510db91639bbe176d82b99" Nov 03 19:30:02.688231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.688206 5209 generic.go:356] "Generic (PLEG): container finished" podID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerID="6b8d914d2e5619545096e323cc8cfa9e7226f95e8b62e1a66150d16a6557926b" exitCode=60 Nov 03 19:30:02.688319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.688291 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerDied","Data":"6b8d914d2e5619545096e323cc8cfa9e7226f95e8b62e1a66150d16a6557926b"} Nov 03 19:30:02.688513 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.688497 5209 scope.go:122] "RemoveContainer" containerID="6b8d914d2e5619545096e323cc8cfa9e7226f95e8b62e1a66150d16a6557926b" Nov 03 19:30:02.689988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.689896 5209 generic.go:356] "Generic (PLEG): container finished" podID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerID="6a982c13464e8e00163e45c5b7065e0292fa1a6d066ae4e283e4d0a3743efbb9" exitCode=60 Nov 03 19:30:02.689988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.689975 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerDied","Data":"6a982c13464e8e00163e45c5b7065e0292fa1a6d066ae4e283e4d0a3743efbb9"} Nov 03 19:30:02.690248 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.690230 5209 scope.go:122] "RemoveContainer" containerID="6a982c13464e8e00163e45c5b7065e0292fa1a6d066ae4e283e4d0a3743efbb9" Nov 03 19:30:02.691691 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.691672 5209 generic.go:356] "Generic (PLEG): container finished" podID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerID="abf32bf433d3eb9b9248ae1612f23fb5078a2981bd48c6e4ae6a754b861b050d" exitCode=60 Nov 03 19:30:02.691966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.691948 5209 scope.go:122] "RemoveContainer" containerID="abf32bf433d3eb9b9248ae1612f23fb5078a2981bd48c6e4ae6a754b861b050d" Nov 03 19:30:02.693360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.693342 5209 generic.go:356] "Generic (PLEG): container finished" podID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerID="ffaff2f1255cc4e923ba2c40e921c64e2ff03e193df846baa1a7b00f006940d9" exitCode=60 Nov 03 19:30:02.693579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.693568 5209 scope.go:122] "RemoveContainer" containerID="ffaff2f1255cc4e923ba2c40e921c64e2ff03e193df846baa1a7b00f006940d9" Nov 03 19:30:02.701319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.701241 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerDied","Data":"abf32bf433d3eb9b9248ae1612f23fb5078a2981bd48c6e4ae6a754b861b050d"} Nov 03 19:30:02.701319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:02.701296 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerDied","Data":"ffaff2f1255cc4e923ba2c40e921c64e2ff03e193df846baa1a7b00f006940d9"} Nov 03 19:30:03.373717 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.373674 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:30:03.378072 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.378041 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.427348 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.427311 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-22ed3b\"/\"default-dockercfg-ghvhm\"" Nov 03 19:30:03.430342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.430309 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-22ed3b\"/\"kube-root-ca.crt\"" Nov 03 19:30:03.430583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.430369 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-22ed3b\"/\"openshift-service-ca.crt\"" Nov 03 19:30:03.433870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.433846 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:30:03.523369 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523260 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh7n\" (UniqueName: \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.523581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523390 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.523711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523574 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.523788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523749 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.524185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523897 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.524185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.523965 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.535968 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.530369 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-22ed3b\"/\"test-model-22ed3b-kserve-self-signed-certs\"" Nov 03 19:30:03.624758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.624690 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.624914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.624784 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.624914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.624853 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.624914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.624910 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.625213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.625039 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh7n\" (UniqueName: \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.625213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.625085 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.625893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.625284 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.625893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.625482 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.625893 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.625744 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.627621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.627602 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.627758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.627741 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.683924 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.683886 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh7n\" (UniqueName: \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\") pod \"test-model-22ed3b-kserve-7c864c78f9-8dkkn\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.691772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.691513 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:03.705582 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.705546 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerStarted","Data":"5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6"} Nov 03 19:30:03.707967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.707931 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerStarted","Data":"4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245"} Nov 03 19:30:03.710876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.710850 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerStarted","Data":"7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c"} Nov 03 19:30:03.714279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.713213 5209 generic.go:356] "Generic (PLEG): container finished" podID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerID="378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452" exitCode=60 Nov 03 19:30:03.714279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.713301 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerDied","Data":"378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452"} Nov 03 19:30:03.714279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.713343 5209 scope.go:122] "RemoveContainer" containerID="6b8d914d2e5619545096e323cc8cfa9e7226f95e8b62e1a66150d16a6557926b" Nov 03 19:30:03.714279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.713740 5209 scope.go:122] "RemoveContainer" containerID="378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452" Nov 03 19:30:03.714279 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:03.714001 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8_odh-ai-gateway-infra(4638a05d-61eb-4a81-b4dc-f47fd4fb8087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" Nov 03 19:30:03.717467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.717442 5209 generic.go:356] "Generic (PLEG): container finished" podID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerID="6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4" exitCode=60 Nov 03 19:30:03.717572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.717493 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerDied","Data":"6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4"} Nov 03 19:30:03.717793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.717774 5209 scope.go:122] "RemoveContainer" containerID="6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4" Nov 03 19:30:03.718086 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:03.718057 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb_odh-ai-gateway-infra(cca0a289-a7ba-4276-b72a-b0572dbe7195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" Nov 03 19:30:03.735597 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:03.735566 5209 scope.go:122] "RemoveContainer" containerID="6a982c13464e8e00163e45c5b7065e0292fa1a6d066ae4e283e4d0a3743efbb9" Nov 03 19:30:04.080751 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.080711 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:30:04.085741 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:30:04.085704 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod628cd800_3f40_4244_9818_7dce28abc006.slice/crio-1a3196e17d9be60587cc323f166a9d0b0b217f242f0c27982ed8249f665953d4 WatchSource:0}: Error finding container 1a3196e17d9be60587cc323f166a9d0b0b217f242f0c27982ed8249f665953d4: Status 404 returned error can't find the container with id 1a3196e17d9be60587cc323f166a9d0b0b217f242f0c27982ed8249f665953d4 Nov 03 19:30:04.726245 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.726206 5209 generic.go:356] "Generic (PLEG): container finished" podID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerID="7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c" exitCode=60 Nov 03 19:30:04.726687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.726289 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerDied","Data":"7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c"} Nov 03 19:30:04.726687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.726353 5209 scope.go:122] "RemoveContainer" containerID="94707f1ae7bc03e7a8d24b8bb1553da00e33ad6ef3510db91639bbe176d82b99" Nov 03 19:30:04.726687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.726575 5209 scope.go:122] "RemoveContainer" containerID="7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c" Nov 03 19:30:04.726917 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:04.726897 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk_odh-ai-gateway-infra(87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" Nov 03 19:30:04.728029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.728005 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" event={"ID":"628cd800-3f40-4244-9818-7dce28abc006","Type":"ContainerStarted","Data":"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5"} Nov 03 19:30:04.728181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.728044 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" event={"ID":"628cd800-3f40-4244-9818-7dce28abc006","Type":"ContainerStarted","Data":"1a3196e17d9be60587cc323f166a9d0b0b217f242f0c27982ed8249f665953d4"} Nov 03 19:30:04.729841 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.729825 5209 scope.go:122] "RemoveContainer" containerID="378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452" Nov 03 19:30:04.730093 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:04.730071 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8_odh-ai-gateway-infra(4638a05d-61eb-4a81-b4dc-f47fd4fb8087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" Nov 03 19:30:04.731267 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.731249 5209 scope.go:122] "RemoveContainer" containerID="6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4" Nov 03 19:30:04.731479 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:04.731461 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb_odh-ai-gateway-infra(cca0a289-a7ba-4276-b72a-b0572dbe7195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" Nov 03 19:30:04.732777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.732755 5209 generic.go:356] "Generic (PLEG): container finished" podID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerID="5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6" exitCode=60 Nov 03 19:30:04.732885 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.732827 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerDied","Data":"5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6"} Nov 03 19:30:04.733007 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.732993 5209 scope.go:122] "RemoveContainer" containerID="5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6" Nov 03 19:30:04.733208 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:04.733188 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5_odh-ai-gateway-infra(5983c826-9d19-4e65-ae77-98484ebde2c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" Nov 03 19:30:04.734799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.734774 5209 generic.go:356] "Generic (PLEG): container finished" podID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerID="4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245" exitCode=60 Nov 03 19:30:04.734905 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.734837 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerDied","Data":"4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245"} Nov 03 19:30:04.735038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.735026 5209 scope.go:122] "RemoveContainer" containerID="4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245" Nov 03 19:30:04.735231 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:04.735215 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w_odh-ai-gateway-infra(cab29859-8aec-40ac-ad30-bd0e8e91ab6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" Nov 03 19:30:04.739444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.739424 5209 scope.go:122] "RemoveContainer" containerID="abf32bf433d3eb9b9248ae1612f23fb5078a2981bd48c6e4ae6a754b861b050d" Nov 03 19:30:04.752633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:04.752616 5209 scope.go:122] "RemoveContainer" containerID="ffaff2f1255cc4e923ba2c40e921c64e2ff03e193df846baa1a7b00f006940d9" Nov 03 19:30:05.272574 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:05.272524 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" podStartSLOduration=2.272512368 podStartE2EDuration="2.272512368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:05.27217269 +0000 UTC m=+2686.683272814" watchObservedRunningTime="2026-11-03 19:30:05.272512368 +0000 UTC m=+2686.683612491" Nov 03 19:30:05.692086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:05.692050 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:05.743807 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:05.743780 5209 scope.go:122] "RemoveContainer" containerID="5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6" Nov 03 19:30:05.744288 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:05.744085 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5_odh-ai-gateway-infra(5983c826-9d19-4e65-ae77-98484ebde2c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" Nov 03 19:30:05.745395 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:05.745373 5209 scope.go:122] "RemoveContainer" containerID="4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245" Nov 03 19:30:05.745622 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:05.745601 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w_odh-ai-gateway-infra(cab29859-8aec-40ac-ad30-bd0e8e91ab6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" Nov 03 19:30:05.746786 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:05.746770 5209 scope.go:122] "RemoveContainer" containerID="7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c" Nov 03 19:30:05.746970 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:05.746954 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk_odh-ai-gateway-infra(87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" Nov 03 19:30:13.692099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:13.692055 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:13.705316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:13.705292 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:13.810021 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:13.809994 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:30:16.699787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:16.699759 5209 scope.go:122] "RemoveContainer" containerID="7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c" Nov 03 19:30:16.700175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:16.699914 5209 scope.go:122] "RemoveContainer" containerID="6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4" Nov 03 19:30:17.692266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.692234 5209 scope.go:122] "RemoveContainer" containerID="4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245" Nov 03 19:30:17.692420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.692395 5209 scope.go:122] "RemoveContainer" containerID="5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6" Nov 03 19:30:17.831777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.831694 5209 generic.go:356] "Generic (PLEG): container finished" podID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerID="9337f2f1bd43b312c324128ec98c5559c2d2651703a1208bce585d71d194f91b" exitCode=60 Nov 03 19:30:17.831777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.831768 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerDied","Data":"9337f2f1bd43b312c324128ec98c5559c2d2651703a1208bce585d71d194f91b"} Nov 03 19:30:17.832279 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.831833 5209 scope.go:122] "RemoveContainer" containerID="6c4d6d47e63b81cda26a30f37af5bfaab11cac2f248860f9a9bff0a4c5f417e4" Nov 03 19:30:17.832337 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.832275 5209 scope.go:122] "RemoveContainer" containerID="9337f2f1bd43b312c324128ec98c5559c2d2651703a1208bce585d71d194f91b" Nov 03 19:30:17.832622 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:17.832594 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb_odh-ai-gateway-infra(cca0a289-a7ba-4276-b72a-b0572dbe7195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" Nov 03 19:30:17.833856 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.833836 5209 generic.go:356] "Generic (PLEG): container finished" podID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerID="f1ee1400182c3761f3fc591d0437f5071d32e6b4d3ddfc5d156f36f825779220" exitCode=60 Nov 03 19:30:17.833948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.833872 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerDied","Data":"f1ee1400182c3761f3fc591d0437f5071d32e6b4d3ddfc5d156f36f825779220"} Nov 03 19:30:17.834193 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.834180 5209 scope.go:122] "RemoveContainer" containerID="f1ee1400182c3761f3fc591d0437f5071d32e6b4d3ddfc5d156f36f825779220" Nov 03 19:30:17.834434 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:17.834417 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk_odh-ai-gateway-infra(87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" Nov 03 19:30:17.845326 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:17.845307 5209 scope.go:122] "RemoveContainer" containerID="7c23f4c32c028d2913443ae04f68e62daacf3eac0cb884c675628aaf19d5bd1c" Nov 03 19:30:18.695517 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.695453 5209 scope.go:122] "RemoveContainer" containerID="378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452" Nov 03 19:30:18.847001 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.846967 5209 generic.go:356] "Generic (PLEG): container finished" podID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerID="d0046d511d9921d5e3221f7128ab739f383a056dbfc737bbb7daf7c8eb72048b" exitCode=60 Nov 03 19:30:18.847464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.847041 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerDied","Data":"d0046d511d9921d5e3221f7128ab739f383a056dbfc737bbb7daf7c8eb72048b"} Nov 03 19:30:18.847464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.847104 5209 scope.go:122] "RemoveContainer" containerID="5b56d121836908adcef49f94ea6ce748081e6be1d29a20f122eb88bbaf57c8d6" Nov 03 19:30:18.847464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.847460 5209 scope.go:122] "RemoveContainer" containerID="d0046d511d9921d5e3221f7128ab739f383a056dbfc737bbb7daf7c8eb72048b" Nov 03 19:30:18.847779 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:18.847758 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5_odh-ai-gateway-infra(5983c826-9d19-4e65-ae77-98484ebde2c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" Nov 03 19:30:18.848873 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.848847 5209 generic.go:356] "Generic (PLEG): container finished" podID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerID="3330d8e3d9e30b018d8d017e02ab5d677b900d5bd5a59f64163341904517d002" exitCode=60 Nov 03 19:30:18.848956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.848901 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerDied","Data":"3330d8e3d9e30b018d8d017e02ab5d677b900d5bd5a59f64163341904517d002"} Nov 03 19:30:18.849277 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.849261 5209 scope.go:122] "RemoveContainer" containerID="3330d8e3d9e30b018d8d017e02ab5d677b900d5bd5a59f64163341904517d002" Nov 03 19:30:18.849543 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:18.849523 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w_odh-ai-gateway-infra(cab29859-8aec-40ac-ad30-bd0e8e91ab6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" Nov 03 19:30:18.860393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.860376 5209 scope.go:122] "RemoveContainer" containerID="4b9111687f20f3023440b95a2ecf3e27600908fd80760d3b85a14c850a130245" Nov 03 19:30:18.951086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:18.951013 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:19.064020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.063980 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:19.121527 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.121502 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:19.256538 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.256514 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:19.270723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.270700 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\") pod \"cca0a289-a7ba-4276-b72a-b0572dbe7195\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " Nov 03 19:30:19.270833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.270805 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\" (UniqueName: \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\") pod \"cca0a289-a7ba-4276-b72a-b0572dbe7195\" (UID: \"cca0a289-a7ba-4276-b72a-b0572dbe7195\") " Nov 03 19:30:19.271031 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.271008 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca" pod "cca0a289-a7ba-4276-b72a-b0572dbe7195" (UID: "cca0a289-a7ba-4276-b72a-b0572dbe7195"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:19.271289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.271272 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca0a289-a7ba-4276-b72a-b0572dbe7195-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.272825 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.272807 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf" pod "cca0a289-a7ba-4276-b72a-b0572dbe7195" (UID: "cca0a289-a7ba-4276-b72a-b0572dbe7195"). InnerVolumeSpecName "kube-api-access-2m6bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:19.372113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.372083 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\" (UniqueName: \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\") pod \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " Nov 03 19:30:19.372275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.372229 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\") pod \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\" (UID: \"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9\") " Nov 03 19:30:19.372632 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.372602 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca" pod "87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" (UID: "87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:19.372724 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.372671 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6bf\" (UniqueName: \"kubernetes.io/projected/cca0a289-a7ba-4276-b72a-b0572dbe7195-kube-api-access-2m6bf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.374186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.374161 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f" pod "87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" (UID: "87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9"). InnerVolumeSpecName "kube-api-access-fvx8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:19.473817 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.473786 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.473817 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.473814 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvx8f\" (UniqueName: \"kubernetes.io/projected/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9-kube-api-access-fvx8f\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.859699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.859622 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" Nov 03 19:30:19.859699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.859664 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk" event={"ID":"87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9","Type":"ContainerDied","Data":"b2a3c3485307056a07d7894178de144fad6eed1efcf10bab8254926000632bc1"} Nov 03 19:30:19.860208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.859726 5209 scope.go:122] "RemoveContainer" containerID="f1ee1400182c3761f3fc591d0437f5071d32e6b4d3ddfc5d156f36f825779220" Nov 03 19:30:19.861630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.861608 5209 generic.go:356] "Generic (PLEG): container finished" podID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerID="5760604279c3b4b498b413e581295a79e1dc621d1d98931844ca52bf392b841d" exitCode=60 Nov 03 19:30:19.861750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.861677 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerDied","Data":"5760604279c3b4b498b413e581295a79e1dc621d1d98931844ca52bf392b841d"} Nov 03 19:30:19.862099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.862081 5209 scope.go:122] "RemoveContainer" containerID="5760604279c3b4b498b413e581295a79e1dc621d1d98931844ca52bf392b841d" Nov 03 19:30:19.862442 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:19.862411 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8_odh-ai-gateway-infra(4638a05d-61eb-4a81-b4dc-f47fd4fb8087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" Nov 03 19:30:19.863046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.863027 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" event={"ID":"cca0a289-a7ba-4276-b72a-b0572dbe7195","Type":"ContainerDied","Data":"0b68e768e592caf17b5d33a0ff110a2c4857ede06f5283598b8e68bd17282d6d"} Nov 03 19:30:19.863134 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.863061 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb" Nov 03 19:30:19.869954 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.869937 5209 scope.go:122] "RemoveContainer" containerID="378c1e2c3cc1ceca7cbf6b07ca9510dee3eb19e0f3471b706620abffc0fd2452" Nov 03 19:30:19.882981 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.882967 5209 scope.go:122] "RemoveContainer" containerID="9337f2f1bd43b312c324128ec98c5559c2d2651703a1208bce585d71d194f91b" Nov 03 19:30:19.969271 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:19.969238 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:20.081653 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.081621 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:20.127077 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.127043 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:20.132099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.132074 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895570-9t6qb"] Nov 03 19:30:20.139028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.139004 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:20.180000 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.179964 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\" (UniqueName: \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\") pod \"5983c826-9d19-4e65-ae77-98484ebde2c7\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " Nov 03 19:30:20.180203 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.180161 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\") pod \"5983c826-9d19-4e65-ae77-98484ebde2c7\" (UID: \"5983c826-9d19-4e65-ae77-98484ebde2c7\") " Nov 03 19:30:20.180587 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.180559 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca" pod "5983c826-9d19-4e65-ae77-98484ebde2c7" (UID: "5983c826-9d19-4e65-ae77-98484ebde2c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:20.186754 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.182688 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh" pod "5983c826-9d19-4e65-ae77-98484ebde2c7" (UID: "5983c826-9d19-4e65-ae77-98484ebde2c7"). InnerVolumeSpecName "kube-api-access-8vjzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:20.231573 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.231543 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:20.238617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.238590 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:20.240639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.240613 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22ed3b-29895570-bctjk"] Nov 03 19:30:20.280882 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.280840 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\") pod \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " Nov 03 19:30:20.281065 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.280949 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\" (UniqueName: \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\") pod \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\" (UID: \"cab29859-8aec-40ac-ad30-bd0e8e91ab6e\") " Nov 03 19:30:20.281264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.281240 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca" pod "cab29859-8aec-40ac-ad30-bd0e8e91ab6e" (UID: "cab29859-8aec-40ac-ad30-bd0e8e91ab6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:20.281520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.281502 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5983c826-9d19-4e65-ae77-98484ebde2c7-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:20.281593 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.281521 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjzh\" (UniqueName: \"kubernetes.io/projected/5983c826-9d19-4e65-ae77-98484ebde2c7-kube-api-access-8vjzh\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:20.281593 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.281535 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:20.283077 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.283053 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk" pod "cab29859-8aec-40ac-ad30-bd0e8e91ab6e" (UID: "cab29859-8aec-40ac-ad30-bd0e8e91ab6e"). InnerVolumeSpecName "kube-api-access-dp5dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:20.382810 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.382724 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5dk\" (UniqueName: \"kubernetes.io/projected/cab29859-8aec-40ac-ad30-bd0e8e91ab6e-kube-api-access-dp5dk\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:20.699375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.699296 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" path="/var/lib/kubelet/pods/87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9/volumes" Nov 03 19:30:20.699716 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.699703 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" path="/var/lib/kubelet/pods/cca0a289-a7ba-4276-b72a-b0572dbe7195/volumes" Nov 03 19:30:20.870077 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.870050 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" Nov 03 19:30:20.870579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.870049 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5" event={"ID":"5983c826-9d19-4e65-ae77-98484ebde2c7","Type":"ContainerDied","Data":"4c60a35413c3b7d136a3185ac4968bd346074a73104bac74ece7f5f2236b143e"} Nov 03 19:30:20.870579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.870189 5209 scope.go:122] "RemoveContainer" containerID="d0046d511d9921d5e3221f7128ab739f383a056dbfc737bbb7daf7c8eb72048b" Nov 03 19:30:20.871559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.871538 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" Nov 03 19:30:20.871668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.871561 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w" event={"ID":"cab29859-8aec-40ac-ad30-bd0e8e91ab6e","Type":"ContainerDied","Data":"8b1e3c9dff6e9c453e757be6d69d47a8a2888a6d822411bdaf3a8ad72f990da4"} Nov 03 19:30:20.880281 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:20.880262 5209 scope.go:122] "RemoveContainer" containerID="3330d8e3d9e30b018d8d017e02ab5d677b900d5bd5a59f64163341904517d002" Nov 03 19:30:21.024451 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.024371 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:21.027173 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.027147 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-73a216-29895570-87sg5"] Nov 03 19:30:21.029306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.029276 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:21.128909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.128878 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:21.132711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.132683 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895570-7lb7w"] Nov 03 19:30:21.176542 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.176520 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:21.194655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.194627 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\" (UniqueName: \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\") pod \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " Nov 03 19:30:21.194787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.194773 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\") pod \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\" (UID: \"4638a05d-61eb-4a81-b4dc-f47fd4fb8087\") " Nov 03 19:30:21.195093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.195071 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca" pod "4638a05d-61eb-4a81-b4dc-f47fd4fb8087" (UID: "4638a05d-61eb-4a81-b4dc-f47fd4fb8087"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:21.196578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.196558 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h" pod "4638a05d-61eb-4a81-b4dc-f47fd4fb8087" (UID: "4638a05d-61eb-4a81-b4dc-f47fd4fb8087"). InnerVolumeSpecName "kube-api-access-hn88h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:21.296039 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.295958 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn88h\" (UniqueName: \"kubernetes.io/projected/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-kube-api-access-hn88h\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:21.296039 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.295987 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4638a05d-61eb-4a81-b4dc-f47fd4fb8087-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:21.885784 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.885746 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" event={"ID":"4638a05d-61eb-4a81-b4dc-f47fd4fb8087","Type":"ContainerDied","Data":"14a9e82af532d855cc4d6d942159783666ce837252cbb0ec544e5e9a7f6d3faf"} Nov 03 19:30:21.886269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.885821 5209 scope.go:122] "RemoveContainer" containerID="5760604279c3b4b498b413e581295a79e1dc621d1d98931844ca52bf392b841d" Nov 03 19:30:21.886269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:21.885760 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8" Nov 03 19:30:22.043319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:22.043290 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:22.089339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:22.089302 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6413ba-29895570-bg8v8"] Nov 03 19:30:22.698345 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:22.698311 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" path="/var/lib/kubelet/pods/4638a05d-61eb-4a81-b4dc-f47fd4fb8087/volumes" Nov 03 19:30:22.698845 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:22.698824 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" path="/var/lib/kubelet/pods/5983c826-9d19-4e65-ae77-98484ebde2c7/volumes" Nov 03 19:30:22.699275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:22.699257 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" path="/var/lib/kubelet/pods/cab29859-8aec-40ac-ad30-bd0e8e91ab6e/volumes" Nov 03 19:30:23.880029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:23.879995 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:30:23.880474 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:23.880220 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" containerID="cri-o://7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e" gracePeriod=30 Nov 03 19:30:25.265458 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:25.265427 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:30:29.151290 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.151236 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:30:29.300564 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300535 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.300758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300614 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.300758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300662 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.300758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300691 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.300758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300723 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.300758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300751 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.301351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300788 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.301351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300852 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.301351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.300886 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\") pod \"76e37712-47d5-4bf1-8748-f5eecb0805b7\" (UID: \"76e37712-47d5-4bf1-8748-f5eecb0805b7\") " Nov 03 19:30:29.301351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301011 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.301351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301026 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.301648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301381 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.301648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301389 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.301648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301438 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.301648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301468 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:29.301648 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.301541 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.303249 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.303222 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "kube-api-access-vbnbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:29.303347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.303233 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.303554 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.303527 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:30:29.303639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.303560 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token" pod "76e37712-47d5-4bf1-8748-f5eecb0805b7" (UID: "76e37712-47d5-4bf1-8748-f5eecb0805b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403018 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403048 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403058 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnbb\" (UniqueName: \"kubernetes.io/projected/76e37712-47d5-4bf1-8748-f5eecb0805b7-kube-api-access-vbnbb\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403069 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403078 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403086 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e37712-47d5-4bf1-8748-f5eecb0805b7-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.403099 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.403094 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e37712-47d5-4bf1-8748-f5eecb0805b7-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.936029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.935991 5209 generic.go:356] "Generic (PLEG): container finished" podID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerID="7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e" exitCode=0 Nov 03 19:30:29.936218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.936048 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" Nov 03 19:30:29.936218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.936080 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerDied","Data":"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e"} Nov 03 19:30:29.936218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.936175 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98" event={"ID":"76e37712-47d5-4bf1-8748-f5eecb0805b7","Type":"ContainerDied","Data":"dfb1827d2a326ec0fc5d5a18ec9ef284f27de878122d05b564b84cef5501a506"} Nov 03 19:30:29.936218 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.936208 5209 scope.go:122] "RemoveContainer" containerID="7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e" Nov 03 19:30:29.947269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.947253 5209 scope.go:122] "RemoveContainer" containerID="287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1" Nov 03 19:30:29.955803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.955783 5209 scope.go:122] "RemoveContainer" containerID="7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e" Nov 03 19:30:29.956053 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:29.956033 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e\": container with ID starting with 7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e not found: ID does not exist" containerID="7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e" Nov 03 19:30:29.956114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.956059 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e"} err="failed to get container status \"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e\": rpc error: code = NotFound desc = could not find container \"7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e\": container with ID starting with 7e188dda47c9622d939dacc0ace2d01b6f91885a0da8a4b6e53f5c1cce161e6e not found: ID does not exist" Nov 03 19:30:29.956114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.956077 5209 scope.go:122] "RemoveContainer" containerID="287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1" Nov 03 19:30:29.956315 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:29.956296 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1\": container with ID starting with 287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1 not found: ID does not exist" containerID="287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1" Nov 03 19:30:29.956356 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:29.956324 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1"} err="failed to get container status \"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1\": rpc error: code = NotFound desc = could not find container \"287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1\": container with ID starting with 287e3eaa9a33e368962dfeac9a40da14ea81cd4e5f4cea4cb89954c91bce1fe1 not found: ID does not exist" Nov 03 19:30:30.139068 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:30.139035 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:30:30.185815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:30.185781 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-c5s98"] Nov 03 19:30:30.698876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:30.698843 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" path="/var/lib/kubelet/pods/76e37712-47d5-4bf1-8748-f5eecb0805b7/volumes" Nov 03 19:30:31.937447 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:30:31.937407 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:30:36.709306 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:36.709260 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:30:36.809468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:36.809435 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:30:41.709679 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:41.709636 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" containerID="cri-o://9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d" gracePeriod=25 Nov 03 19:30:41.810822 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:41.810766 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" containerID="cri-o://e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e" gracePeriod=25 Nov 03 19:30:47.758675 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:47.758629 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:30:47.759240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:47.758631 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:30:51.051235 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:51.051187 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:30:57.759152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:57.759099 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:30:57.759604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:30:57.759165 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:31:01.199806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:01.199768 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:31:01.303327 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:01.303295 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:31:01.941090 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:01.941051 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:31:01.980324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:01.980297 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:31:02.062964 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.062935 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:31:02.101849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.101819 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\") pod \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " Nov 03 19:31:02.102018 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.101929 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\" (UniqueName: \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\") pod \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\" (UID: \"2a0a985a-baa9-426b-b5c5-d4d21a64228a\") " Nov 03 19:31:02.102349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.102320 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume" pod "2a0a985a-baa9-426b-b5c5-d4d21a64228a" (UID: "2a0a985a-baa9-426b-b5c5-d4d21a64228a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:02.102487 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.102466 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a985a-baa9-426b-b5c5-d4d21a64228a-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:02.104373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.104346 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq" pod "2a0a985a-baa9-426b-b5c5-d4d21a64228a" (UID: "2a0a985a-baa9-426b-b5c5-d4d21a64228a"). InnerVolumeSpecName "kube-api-access-bwvkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:02.128816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.128783 5209 generic.go:356] "Generic (PLEG): container finished" podID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerID="e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e" exitCode=0 Nov 03 19:31:02.129027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.128827 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" Nov 03 19:31:02.129027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.128866 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" event={"ID":"873a9b12-181c-49ed-9a18-2e27cc380fd3","Type":"ContainerDied","Data":"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e"} Nov 03 19:31:02.129027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.128931 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk" event={"ID":"873a9b12-181c-49ed-9a18-2e27cc380fd3","Type":"ContainerDied","Data":"fc030007b939960b674c51f5556ae2efec3fbe50d3352ec474554c8ce5f33974"} Nov 03 19:31:02.129027 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.128962 5209 scope.go:122] "RemoveContainer" containerID="e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e" Nov 03 19:31:02.130475 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.130452 5209 generic.go:356] "Generic (PLEG): container finished" podID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerID="9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d" exitCode=0 Nov 03 19:31:02.130584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.130487 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" event={"ID":"2a0a985a-baa9-426b-b5c5-d4d21a64228a","Type":"ContainerDied","Data":"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d"} Nov 03 19:31:02.130584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.130502 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" Nov 03 19:31:02.130584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.130518 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n" event={"ID":"2a0a985a-baa9-426b-b5c5-d4d21a64228a","Type":"ContainerDied","Data":"2d89693c43b2ce3463e2373def35500276936845fed010e1086b550eab88ae09"} Nov 03 19:31:02.142641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.140766 5209 scope.go:122] "RemoveContainer" containerID="e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e" Nov 03 19:31:02.142641 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:02.141280 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e\": container with ID starting with e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e not found: ID does not exist" containerID="e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e" Nov 03 19:31:02.142641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.141314 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e"} err="failed to get container status \"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e\": rpc error: code = NotFound desc = could not find container \"e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e\": container with ID starting with e054bdac6c8a30110d11087c3ae3528ede9a3f9ed695bd7aa49b2787daedf72e not found: ID does not exist" Nov 03 19:31:02.142641 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.141339 5209 scope.go:122] "RemoveContainer" containerID="9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d" Nov 03 19:31:02.150837 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.150820 5209 scope.go:122] "RemoveContainer" containerID="9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d" Nov 03 19:31:02.151092 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:02.151076 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d\": container with ID starting with 9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d not found: ID does not exist" containerID="9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d" Nov 03 19:31:02.151154 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.151099 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d"} err="failed to get container status \"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d\": rpc error: code = NotFound desc = could not find container \"9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d\": container with ID starting with 9a3dc39c96f36b4eace9d5224011e89acda23ab00db87b869df7b0200d8bf21d not found: ID does not exist" Nov 03 19:31:02.202976 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.202941 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\" (UniqueName: \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\") pod \"873a9b12-181c-49ed-9a18-2e27cc380fd3\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " Nov 03 19:31:02.203435 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.203092 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\") pod \"873a9b12-181c-49ed-9a18-2e27cc380fd3\" (UID: \"873a9b12-181c-49ed-9a18-2e27cc380fd3\") " Nov 03 19:31:02.203520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.203495 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume" pod "873a9b12-181c-49ed-9a18-2e27cc380fd3" (UID: "873a9b12-181c-49ed-9a18-2e27cc380fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:02.204084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.204052 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvkq\" (UniqueName: \"kubernetes.io/projected/2a0a985a-baa9-426b-b5c5-d4d21a64228a-kube-api-access-bwvkq\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:02.204084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.204078 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873a9b12-181c-49ed-9a18-2e27cc380fd3-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:02.205723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.205698 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g" pod "873a9b12-181c-49ed-9a18-2e27cc380fd3" (UID: "873a9b12-181c-49ed-9a18-2e27cc380fd3"). InnerVolumeSpecName "kube-api-access-b2c6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:02.304715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.304635 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2c6g\" (UniqueName: \"kubernetes.io/projected/873a9b12-181c-49ed-9a18-2e27cc380fd3-kube-api-access-b2c6g\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:02.365630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.365589 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:31:02.369351 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.369313 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332dac-65dc8565d-8kz6n"] Nov 03 19:31:02.543548 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.543512 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:31:02.547369 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.547340 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332dac-6f469cfbc9-5k7rk"] Nov 03 19:31:02.732793 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.732755 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" path="/var/lib/kubelet/pods/2a0a985a-baa9-426b-b5c5-d4d21a64228a/volumes" Nov 03 19:31:02.734637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:02.733455 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" path="/var/lib/kubelet/pods/873a9b12-181c-49ed-9a18-2e27cc380fd3/volumes" Nov 03 19:31:06.201113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.201074 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" podUID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" containerName="payload-processing" containerID="cri-o://8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea" gracePeriod=25 Nov 03 19:31:06.304405 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.304358 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" containerID="cri-o://f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae" gracePeriod=25 Nov 03 19:31:06.454639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.454576 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:31:06.555854 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.555817 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\" (UniqueName: \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\") pod \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " Nov 03 19:31:06.556026 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.555897 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\") pod \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\" (UID: \"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5\") " Nov 03 19:31:06.556263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.556242 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume" pod "c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" (UID: "c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:06.557970 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.557951 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl" pod "c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" (UID: "c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5"). InnerVolumeSpecName "kube-api-access-b7jjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:06.657289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.657256 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jjl\" (UniqueName: \"kubernetes.io/projected/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-kube-api-access-b7jjl\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:06.657289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:06.657284 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:07.167093 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.167058 5209 generic.go:356] "Generic (PLEG): container finished" podID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" containerID="8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea" exitCode=0 Nov 03 19:31:07.167286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.167108 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" Nov 03 19:31:07.167286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.167152 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" event={"ID":"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5","Type":"ContainerDied","Data":"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea"} Nov 03 19:31:07.167286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.167209 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj" event={"ID":"c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5","Type":"ContainerDied","Data":"6e7060d8168df5d729e5de0a8b46e357d8fbb363b936167ed4eb6f135fdfa26d"} Nov 03 19:31:07.167286 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.167235 5209 scope.go:122] "RemoveContainer" containerID="8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea" Nov 03 19:31:07.176852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.176818 5209 scope.go:122] "RemoveContainer" containerID="8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea" Nov 03 19:31:07.177165 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:07.177146 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea\": container with ID starting with 8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea not found: ID does not exist" containerID="8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea" Nov 03 19:31:07.177250 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.177177 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea"} err="failed to get container status \"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea\": rpc error: code = NotFound desc = could not find container \"8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea\": container with ID starting with 8a24f840d9407c4249b866afd160f00dd8e0a433977a0486654f066b846e8cea not found: ID does not exist" Nov 03 19:31:07.323290 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.323255 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:31:07.370105 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:07.370070 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6413ba-d8c95df7d-2d4lj"] Nov 03 19:31:08.702209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:08.702172 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" path="/var/lib/kubelet/pods/c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5/volumes" Nov 03 19:31:09.700421 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:09.700381 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:31:19.701210 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:19.701147 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:31:25.455953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:25.455916 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:31:26.010400 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.010367 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:31:26.010595 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.010573 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" containerID="cri-o://f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e" gracePeriod=30 Nov 03 19:31:26.556299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.556273 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:31:26.683043 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.683012 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\" (UniqueName: \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\") pod \"886455ec-0ba7-4333-b783-3606386adf82\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " Nov 03 19:31:26.683280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.683152 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\") pod \"886455ec-0ba7-4333-b783-3606386adf82\" (UID: \"886455ec-0ba7-4333-b783-3606386adf82\") " Nov 03 19:31:26.683584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.683559 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume" pod "886455ec-0ba7-4333-b783-3606386adf82" (UID: "886455ec-0ba7-4333-b783-3606386adf82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:26.685064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.685043 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k" pod "886455ec-0ba7-4333-b783-3606386adf82" (UID: "886455ec-0ba7-4333-b783-3606386adf82"). InnerVolumeSpecName "kube-api-access-pbr9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:26.785621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.785577 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/886455ec-0ba7-4333-b783-3606386adf82-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:26.785621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:26.785606 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbr9k\" (UniqueName: \"kubernetes.io/projected/886455ec-0ba7-4333-b783-3606386adf82-kube-api-access-pbr9k\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:27.297734 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.297699 5209 generic.go:356] "Generic (PLEG): container finished" podID="886455ec-0ba7-4333-b783-3606386adf82" containerID="f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae" exitCode=0 Nov 03 19:31:27.297914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.297750 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" Nov 03 19:31:27.297914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.297783 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" event={"ID":"886455ec-0ba7-4333-b783-3606386adf82","Type":"ContainerDied","Data":"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae"} Nov 03 19:31:27.297914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.297842 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl" event={"ID":"886455ec-0ba7-4333-b783-3606386adf82","Type":"ContainerDied","Data":"5944df31aab60d9dac23792b40394b8e69ef43f93b1e6a067a4fcdea6819391f"} Nov 03 19:31:27.297914 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.297864 5209 scope.go:122] "RemoveContainer" containerID="f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae" Nov 03 19:31:27.311087 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.311059 5209 scope.go:122] "RemoveContainer" containerID="f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae" Nov 03 19:31:27.311388 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:27.311356 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae\": container with ID starting with f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae not found: ID does not exist" containerID="f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae" Nov 03 19:31:27.311476 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.311390 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae"} err="failed to get container status \"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae\": rpc error: code = NotFound desc = could not find container \"f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae\": container with ID starting with f357c604f941c5c334110f798be078651c28d8ca373f9ef2bb90f08e55ddddae not found: ID does not exist" Nov 03 19:31:27.451757 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.451721 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:31:27.498197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:27.498171 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6413ba-5b8684bc59-ftqfl"] Nov 03 19:31:28.699282 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:28.699249 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886455ec-0ba7-4333-b783-3606386adf82" path="/var/lib/kubelet/pods/886455ec-0ba7-4333-b783-3606386adf82/volumes" Nov 03 19:31:31.275632 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.275580 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:31:31.323794 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.323768 5209 generic.go:356] "Generic (PLEG): container finished" podID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerID="f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e" exitCode=0 Nov 03 19:31:31.323923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.323816 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" Nov 03 19:31:31.323923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.323859 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerDied","Data":"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e"} Nov 03 19:31:31.323923 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.323919 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v" event={"ID":"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4","Type":"ContainerDied","Data":"69da8a4158802fd875b73b48107a8b4473ef4588add6b422ba911e53e23c4376"} Nov 03 19:31:31.324028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.323949 5209 scope.go:122] "RemoveContainer" containerID="f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e" Nov 03 19:31:31.333007 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.332991 5209 scope.go:122] "RemoveContainer" containerID="a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b" Nov 03 19:31:31.341690 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.341664 5209 scope.go:122] "RemoveContainer" containerID="f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e" Nov 03 19:31:31.341949 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:31.341932 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e\": container with ID starting with f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e not found: ID does not exist" containerID="f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e" Nov 03 19:31:31.341993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.341961 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e"} err="failed to get container status \"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e\": rpc error: code = NotFound desc = could not find container \"f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e\": container with ID starting with f467455a1604a80db4dff5bb2a937f9e7fb3715a101c8cdc905e31f8113d644e not found: ID does not exist" Nov 03 19:31:31.341993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.341980 5209 scope.go:122] "RemoveContainer" containerID="a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b" Nov 03 19:31:31.342245 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:31.342224 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b\": container with ID starting with a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b not found: ID does not exist" containerID="a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b" Nov 03 19:31:31.342297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.342251 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b"} err="failed to get container status \"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b\": rpc error: code = NotFound desc = could not find container \"a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b\": container with ID starting with a23d0543ff5a5c15b1592690a96fd3d2179be3bf7d74ca36feb3d24ca279191b not found: ID does not exist" Nov 03 19:31:31.439812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.439782 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.439870 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.439903 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.439935 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.439985 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440164 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440199 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440240 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440233 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440401 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440284 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\") pod \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\" (UID: \"f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4\") " Nov 03 19:31:31.440401 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440349 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:31.440499 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440433 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:31.441046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440763 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.441046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440789 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.441046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440797 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:31.441046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.440897 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:31.441046 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.441013 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:31.445560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.445534 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:31.445687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.445568 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:31.445687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.445587 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "kube-api-access-wpsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:31.445801 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.445732 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo" pod "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" (UID: "f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541438 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsbw\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-kube-api-access-wpsbw\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541466 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541476 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541484 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541493 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541501 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.541512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.541510 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:31.740709 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.740676 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:31:31.744567 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.744540 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6413ba-openshift-default-6bff7587f7-2sz4v"] Nov 03 19:31:31.915331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.915296 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:31:31.915574 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:31.915532 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" containerID="cri-o://6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e" gracePeriod=30 Nov 03 19:31:31.953730 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:31.953688 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:31:32.627550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:32.627518 5209 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:31:32.627953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:32.627576 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:31:32.697927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:32.697894 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" path="/var/lib/kubelet/pods/f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4/volumes" Nov 03 19:31:37.186956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.186927 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:31:37.198294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198248 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198288 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198328 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198444 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198536 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198610 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198694 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198659 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198713 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.198808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198733 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.198808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198763 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:37.198808 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198784 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.198969 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.198818 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\") pod \"f91bb60f-afed-4f87-a192-283e15d7ad31\" (UID: \"f91bb60f-afed-4f87-a192-283e15d7ad31\") " Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199050 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199291 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199406 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91bb60f-afed-4f87-a192-283e15d7ad31-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199426 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199441 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199457 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.199495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.199472 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.200650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.200625 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:31:37.200788 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.200768 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:37.200862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.200831 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "kube-api-access-sf889". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:37.200862 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.200837 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy" pod "f91bb60f-afed-4f87-a192-283e15d7ad31" (UID: "f91bb60f-afed-4f87-a192-283e15d7ad31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.300468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.300443 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.300468 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.300466 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.300618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.300477 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf889\" (UniqueName: \"kubernetes.io/projected/f91bb60f-afed-4f87-a192-283e15d7ad31-kube-api-access-sf889\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.300618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.300486 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91bb60f-afed-4f87-a192-283e15d7ad31-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.362239 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.362206 5209 generic.go:356] "Generic (PLEG): container finished" podID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerID="6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e" exitCode=0 Nov 03 19:31:37.362425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.362264 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" Nov 03 19:31:37.362425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.362277 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerDied","Data":"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e"} Nov 03 19:31:37.362425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.362328 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4" event={"ID":"f91bb60f-afed-4f87-a192-283e15d7ad31","Type":"ContainerDied","Data":"2c206fef5debacec43e0dbba9350aa651f15a71658cca10f2cc2be9a874cf879"} Nov 03 19:31:37.362425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.362358 5209 scope.go:122] "RemoveContainer" containerID="6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e" Nov 03 19:31:37.373199 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.373177 5209 scope.go:122] "RemoveContainer" containerID="110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7" Nov 03 19:31:37.382575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.382553 5209 scope.go:122] "RemoveContainer" containerID="6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e" Nov 03 19:31:37.382851 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:37.382824 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e\": container with ID starting with 6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e not found: ID does not exist" containerID="6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e" Nov 03 19:31:37.382911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.382863 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e"} err="failed to get container status \"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e\": rpc error: code = NotFound desc = could not find container \"6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e\": container with ID starting with 6e79aebc11e902c189b9c77aedf5e04f7482ef47f95058c46eba9251d1361d5e not found: ID does not exist" Nov 03 19:31:37.382911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.382891 5209 scope.go:122] "RemoveContainer" containerID="110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7" Nov 03 19:31:37.383234 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:37.383202 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7\": container with ID starting with 110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7 not found: ID does not exist" containerID="110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7" Nov 03 19:31:37.383347 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.383241 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7"} err="failed to get container status \"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7\": rpc error: code = NotFound desc = could not find container \"110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7\": container with ID starting with 110394852af94af4b230bc1a4f26643820a53c057ed28bd8925609e5af36c5a7 not found: ID does not exist" Nov 03 19:31:37.519227 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.519149 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:31:37.530755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:37.530730 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-7bpr4"] Nov 03 19:31:38.310639 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:38.310604 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:31:38.700900 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:38.700859 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" path="/var/lib/kubelet/pods/f91bb60f-afed-4f87-a192-283e15d7ad31/volumes" Nov 03 19:31:40.963228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:40.963192 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:31:40.963731 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:40.963450 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" containerID="cri-o://6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af" gracePeriod=30 Nov 03 19:31:43.311984 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:43.311928 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" containerID="cri-o://2cc0dda58517da59665cac3ffec383c81e1070694568c777b2906c62212fc8b5" gracePeriod=25 Nov 03 19:31:46.228927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.228856 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:31:46.299947 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.299914 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.299972 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300065 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300151 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300263 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300207 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300275 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300333 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300379 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300371 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300411 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\") pod \"68800080-3f84-4b48-a0ac-31ab9d9b5119\" (UID: \"68800080-3f84-4b48-a0ac-31ab9d9b5119\") " Nov 03 19:31:46.300585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300523 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:46.300796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300770 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:46.300796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300783 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:46.300992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300953 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:46.300992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300972 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.301079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.300995 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.301079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.301010 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.301079 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.301026 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:46.302338 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.302316 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:31:46.302425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.302341 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:46.302620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.302594 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "kube-api-access-jkdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:46.302620 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.302609 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy" pod "68800080-3f84-4b48-a0ac-31ab9d9b5119" (UID: "68800080-3f84-4b48-a0ac-31ab9d9b5119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:46.401846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401806 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.401846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401835 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.401846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401845 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68800080-3f84-4b48-a0ac-31ab9d9b5119-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.401846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401854 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68800080-3f84-4b48-a0ac-31ab9d9b5119-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.402135 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401864 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdqv\" (UniqueName: \"kubernetes.io/projected/68800080-3f84-4b48-a0ac-31ab9d9b5119-kube-api-access-jkdqv\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.402135 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.401874 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68800080-3f84-4b48-a0ac-31ab9d9b5119-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:46.414484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.414449 5209 generic.go:356] "Generic (PLEG): container finished" podID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerID="6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af" exitCode=0 Nov 03 19:31:46.414627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.414520 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" Nov 03 19:31:46.414627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.414535 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" event={"ID":"68800080-3f84-4b48-a0ac-31ab9d9b5119","Type":"ContainerDied","Data":"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af"} Nov 03 19:31:46.414627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.414622 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425" event={"ID":"68800080-3f84-4b48-a0ac-31ab9d9b5119","Type":"ContainerDied","Data":"6ec4c9cdefac8d07135bf02fa8a9690665f8ad2e4ed72d7483a4c6b17fff4948"} Nov 03 19:31:46.414762 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.414647 5209 scope.go:122] "RemoveContainer" containerID="6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af" Nov 03 19:31:46.424961 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.424943 5209 scope.go:122] "RemoveContainer" containerID="6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af" Nov 03 19:31:46.425381 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:31:46.425312 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af\": container with ID starting with 6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af not found: ID does not exist" containerID="6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af" Nov 03 19:31:46.425381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.425338 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af"} err="failed to get container status \"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af\": rpc error: code = NotFound desc = could not find container \"6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af\": container with ID starting with 6a80360ba9f0f99ca8aaa307bd78ee51b2168c7d7e02caa8e6fdfa90d9c202af not found: ID does not exist" Nov 03 19:31:46.569699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.569625 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:31:46.574241 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.574217 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2m425"] Nov 03 19:31:46.699802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:46.699770 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" path="/var/lib/kubelet/pods/68800080-3f84-4b48-a0ac-31ab9d9b5119/volumes" Nov 03 19:31:50.172840 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:31:50.172800 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:32:00.172772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:00.172678 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:32:01.936930 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:32:01.936882 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:32:03.517181 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.517140 5209 generic.go:356] "Generic (PLEG): container finished" podID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerID="2cc0dda58517da59665cac3ffec383c81e1070694568c777b2906c62212fc8b5" exitCode=0 Nov 03 19:32:03.517571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.517211 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" event={"ID":"ffc7f747-2e30-4cb4-925a-a16ad1167144","Type":"ContainerDied","Data":"2cc0dda58517da59665cac3ffec383c81e1070694568c777b2906c62212fc8b5"} Nov 03 19:32:03.563307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.563277 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:32:03.570667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.570617 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:32:03.680559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.680524 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\" (UniqueName: \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\") pod \"ffc7f747-2e30-4cb4-925a-a16ad1167144\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " Nov 03 19:32:03.680802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.680572 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\") pod \"ffc7f747-2e30-4cb4-925a-a16ad1167144\" (UID: \"ffc7f747-2e30-4cb4-925a-a16ad1167144\") " Nov 03 19:32:03.681058 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.681013 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume" pod "ffc7f747-2e30-4cb4-925a-a16ad1167144" (UID: "ffc7f747-2e30-4cb4-925a-a16ad1167144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:03.683993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.683948 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq" pod "ffc7f747-2e30-4cb4-925a-a16ad1167144" (UID: "ffc7f747-2e30-4cb4-925a-a16ad1167144"). InnerVolumeSpecName "kube-api-access-xdlgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:03.781772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.781738 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdlgq\" (UniqueName: \"kubernetes.io/projected/ffc7f747-2e30-4cb4-925a-a16ad1167144-kube-api-access-xdlgq\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:03.781772 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:03.781767 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc7f747-2e30-4cb4-925a-a16ad1167144-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:04.524998 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:04.524961 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" Nov 03 19:32:04.524998 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:04.524986 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j" event={"ID":"ffc7f747-2e30-4cb4-925a-a16ad1167144","Type":"ContainerDied","Data":"d80b27707348f35d557ba12a78088148376611f859de7371504f526bef7bf80b"} Nov 03 19:32:04.525602 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:04.525064 5209 scope.go:122] "RemoveContainer" containerID="2cc0dda58517da59665cac3ffec383c81e1070694568c777b2906c62212fc8b5" Nov 03 19:32:04.679381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:04.679346 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:32:04.725776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:04.725742 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c3e530-76d8558695-8xc4j"] Nov 03 19:32:06.371478 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:06.371448 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:32:06.700861 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:06.700782 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" path="/var/lib/kubelet/pods/ffc7f747-2e30-4cb4-925a-a16ad1167144/volumes" Nov 03 19:32:21.391625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.391568 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerName="main" containerID="cri-o://bdac714a86035bef0dce4c8563c05d792cf7ded4a34a1b36430061bf06616325" gracePeriod=45 Nov 03 19:32:21.626951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.626920 5209 generic.go:356] "Generic (PLEG): container finished" podID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerID="bdac714a86035bef0dce4c8563c05d792cf7ded4a34a1b36430061bf06616325" exitCode=0 Nov 03 19:32:21.627183 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.626959 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" event={"ID":"af95000b-df6e-4a58-a114-f4e5ca7a5382","Type":"ContainerDied","Data":"bdac714a86035bef0dce4c8563c05d792cf7ded4a34a1b36430061bf06616325"} Nov 03 19:32:21.645537 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.645479 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:32:21.772644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772616 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.772848 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772701 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\" (UniqueName: \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.772848 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772766 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.772848 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772817 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.773069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772894 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\" (UniqueName: \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.773069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772924 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:21.773069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.772982 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\") pod \"af95000b-df6e-4a58-a114-f4e5ca7a5382\" (UID: \"af95000b-df6e-4a58-a114-f4e5ca7a5382\") " Nov 03 19:32:21.773276 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.773151 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:21.773339 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.773277 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:21.773393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.773355 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:21.773393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.773371 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:21.773393 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.773381 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:21.775073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.775049 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:21.775320 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.775304 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:21.775406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.775354 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7" pod "af95000b-df6e-4a58-a114-f4e5ca7a5382" (UID: "af95000b-df6e-4a58-a114-f4e5ca7a5382"). InnerVolumeSpecName "kube-api-access-bmlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:21.874646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.874607 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af95000b-df6e-4a58-a114-f4e5ca7a5382-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:21.874646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.874641 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmlq7\" (UniqueName: \"kubernetes.io/projected/af95000b-df6e-4a58-a114-f4e5ca7a5382-kube-api-access-bmlq7\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:21.874646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:21.874655 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af95000b-df6e-4a58-a114-f4e5ca7a5382-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:22.596213 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.596167 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Nov 03 19:32:22.633771 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.633736 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" Nov 03 19:32:22.633980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.633755 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l" event={"ID":"af95000b-df6e-4a58-a114-f4e5ca7a5382","Type":"ContainerDied","Data":"c2821f133b1d1e5371cca71ddd8e5139fb68b1f63c4c6299edd3b39ccc645967"} Nov 03 19:32:22.633980 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.633856 5209 scope.go:122] "RemoveContainer" containerID="bdac714a86035bef0dce4c8563c05d792cf7ded4a34a1b36430061bf06616325" Nov 03 19:32:22.789142 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.789065 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:32:22.791644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.791623 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-73a216/test-model-73a216-kserve-7795f655c5-nbr2l"] Nov 03 19:32:22.997806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:22.997721 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:32:24.698918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:24.698884 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" path="/var/lib/kubelet/pods/af95000b-df6e-4a58-a114-f4e5ca7a5382/volumes" Nov 03 19:32:31.279081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:31.279042 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:32:31.938203 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:32:31.938160 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:32:38.018342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.018299 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" podUID="628cd800-3f40-4244-9818-7dce28abc006" containerName="main" containerID="cri-o://018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5" gracePeriod=45 Nov 03 19:32:38.284089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.284017 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:32:38.444427 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444398 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444433 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444454 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444477 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444544 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\" (UniqueName: \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444660 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\" (UniqueName: \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\") pod \"628cd800-3f40-4244-9818-7dce28abc006\" (UID: \"628cd800-3f40-4244-9818-7dce28abc006\") " Nov 03 19:32:38.444843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444676 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:38.444843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444714 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:38.444843 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.444778 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:38.445081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.445062 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.445158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.445089 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.445158 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.445103 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.446786 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.446754 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:38.446900 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.446825 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:38.446900 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.446841 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n" pod "628cd800-3f40-4244-9818-7dce28abc006" (UID: "628cd800-3f40-4244-9818-7dce28abc006"). InnerVolumeSpecName "kube-api-access-lhh7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:38.546703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.546624 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628cd800-3f40-4244-9818-7dce28abc006-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.546703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.546652 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhh7n\" (UniqueName: \"kubernetes.io/projected/628cd800-3f40-4244-9818-7dce28abc006-kube-api-access-lhh7n\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.546703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.546675 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628cd800-3f40-4244-9818-7dce28abc006-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:38.726228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.726195 5209 generic.go:356] "Generic (PLEG): container finished" podID="628cd800-3f40-4244-9818-7dce28abc006" containerID="018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5" exitCode=0 Nov 03 19:32:38.726359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.726238 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" Nov 03 19:32:38.726359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.726276 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" event={"ID":"628cd800-3f40-4244-9818-7dce28abc006","Type":"ContainerDied","Data":"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5"} Nov 03 19:32:38.726359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.726314 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn" event={"ID":"628cd800-3f40-4244-9818-7dce28abc006","Type":"ContainerDied","Data":"1a3196e17d9be60587cc323f166a9d0b0b217f242f0c27982ed8249f665953d4"} Nov 03 19:32:38.726359 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.726333 5209 scope.go:122] "RemoveContainer" containerID="018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5" Nov 03 19:32:38.735069 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.735036 5209 scope.go:122] "RemoveContainer" containerID="018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5" Nov 03 19:32:38.735396 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:32:38.735375 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5\": container with ID starting with 018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5 not found: ID does not exist" containerID="018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5" Nov 03 19:32:38.735465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.735406 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5"} err="failed to get container status \"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5\": rpc error: code = NotFound desc = could not find container \"018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5\": container with ID starting with 018cd82e2357e7cd7d22667da8e0545a57df46ed82ac8f7e37e78b7e41761de5 not found: ID does not exist" Nov 03 19:32:38.879873 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.879839 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:32:38.895796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:38.895761 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-22ed3b/test-model-22ed3b-kserve-7c864c78f9-8dkkn"] Nov 03 19:32:40.697089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:32:40.697056 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628cd800-3f40-4244-9818-7dce28abc006" path="/var/lib/kubelet/pods/628cd800-3f40-4244-9818-7dce28abc006/volumes" Nov 03 19:33:01.937069 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:33:01.937029 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:33:08.474795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:08.474762 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:33:10.457189 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:10.457148 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:33:10.457672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:10.457400 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" containerID="cri-o://d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392" gracePeriod=30 Nov 03 19:33:15.715315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.715290 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:33:15.819776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819691 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819748 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819799 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819840 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819922 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819955 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.819996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.819986 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.820302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820078 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.820302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820148 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\") pod \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\" (UID: \"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb\") " Nov 03 19:33:15.820302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820182 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:15.820302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820250 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:15.820302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820268 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:15.820560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820471 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:15.820560 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820523 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:15.820649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820633 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.820695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820652 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.820695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820667 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.820795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820719 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.820795 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.820735 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.822236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.822210 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:15.822355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.822278 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:15.822449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.822433 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:33:15.822501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.822482 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp" pod "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" (UID: "c93f4c25-cc2a-4123-90c7-98eec6b3a0cb"). InnerVolumeSpecName "kube-api-access-8srzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:15.921625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.921590 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srzp\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-kube-api-access-8srzp\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.921625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.921618 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.921625 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.921629 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.921846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.921637 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.930537 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.930509 5209 generic.go:356] "Generic (PLEG): container finished" podID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerID="d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392" exitCode=0 Nov 03 19:33:15.930699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.930551 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" Nov 03 19:33:15.930699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.930591 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" event={"ID":"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb","Type":"ContainerDied","Data":"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392"} Nov 03 19:33:15.930699 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.930680 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2" event={"ID":"c93f4c25-cc2a-4123-90c7-98eec6b3a0cb","Type":"ContainerDied","Data":"20feffd7b2c6f31292a136198b316af43931fcfbd0df57214d703fc52c28fd6e"} Nov 03 19:33:15.930870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.930709 5209 scope.go:122] "RemoveContainer" containerID="d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392" Nov 03 19:33:15.940744 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.940729 5209 scope.go:122] "RemoveContainer" containerID="d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392" Nov 03 19:33:15.941048 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:33:15.941023 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392\": container with ID starting with d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392 not found: ID does not exist" containerID="d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392" Nov 03 19:33:15.941105 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:15.941057 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392"} err="failed to get container status \"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392\": rpc error: code = NotFound desc = could not find container \"d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392\": container with ID starting with d596fcd2351fe62f124028a5282c2cce29ade1938253792358a73f8a7b671392 not found: ID does not exist" Nov 03 19:33:16.084877 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:16.084847 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:33:16.089081 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:16.089053 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-hkjl2"] Nov 03 19:33:16.698047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:16.698011 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" path="/var/lib/kubelet/pods/c93f4c25-cc2a-4123-90c7-98eec6b3a0cb/volumes" Nov 03 19:33:29.451074 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:29.450989 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:33:31.936653 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:33:31.936607 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:33:34.235677 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:34.235646 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:33:34.336686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:34.336652 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:33:39.236204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.236148 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" podUID="4f0a983e-24ee-4862-bd44-796db6c6079b" containerName="payload-processing" containerID="cri-o://37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c" gracePeriod=25 Nov 03 19:33:39.336972 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.336924 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" podUID="426d5439-a51f-41c6-99c5-5f914803205f" containerName="payload-pre-processing" containerID="cri-o://64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17" gracePeriod=25 Nov 03 19:33:39.490558 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.490503 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:33:39.550815 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.550780 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\") pod \"4f0a983e-24ee-4862-bd44-796db6c6079b\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " Nov 03 19:33:39.550960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.550943 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\" (UniqueName: \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\") pod \"4f0a983e-24ee-4862-bd44-796db6c6079b\" (UID: \"4f0a983e-24ee-4862-bd44-796db6c6079b\") " Nov 03 19:33:39.551207 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.551179 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume" pod "4f0a983e-24ee-4862-bd44-796db6c6079b" (UID: "4f0a983e-24ee-4862-bd44-796db6c6079b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:39.551385 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.551365 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0a983e-24ee-4862-bd44-796db6c6079b-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:39.553032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.553004 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k" pod "4f0a983e-24ee-4862-bd44-796db6c6079b" (UID: "4f0a983e-24ee-4862-bd44-796db6c6079b"). InnerVolumeSpecName "kube-api-access-bxt8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:39.601745 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.601715 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:33:39.651789 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.651760 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\" (UniqueName: \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\") pod \"426d5439-a51f-41c6-99c5-5f914803205f\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " Nov 03 19:33:39.651982 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.651824 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\") pod \"426d5439-a51f-41c6-99c5-5f914803205f\" (UID: \"426d5439-a51f-41c6-99c5-5f914803205f\") " Nov 03 19:33:39.652138 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.652076 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt8k\" (UniqueName: \"kubernetes.io/projected/4f0a983e-24ee-4862-bd44-796db6c6079b-kube-api-access-bxt8k\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:39.652280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.652259 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume" pod "426d5439-a51f-41c6-99c5-5f914803205f" (UID: "426d5439-a51f-41c6-99c5-5f914803205f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:39.653956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.653935 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb" pod "426d5439-a51f-41c6-99c5-5f914803205f" (UID: "426d5439-a51f-41c6-99c5-5f914803205f"). InnerVolumeSpecName "kube-api-access-p5rbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:39.753297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.753230 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rbb\" (UniqueName: \"kubernetes.io/projected/426d5439-a51f-41c6-99c5-5f914803205f-kube-api-access-p5rbb\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:39.753297 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:39.753256 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426d5439-a51f-41c6-99c5-5f914803205f-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:40.061748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.061659 5209 generic.go:356] "Generic (PLEG): container finished" podID="4f0a983e-24ee-4862-bd44-796db6c6079b" containerID="37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c" exitCode=0 Nov 03 19:33:40.061748 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.061707 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" Nov 03 19:33:40.061996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.061744 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" event={"ID":"4f0a983e-24ee-4862-bd44-796db6c6079b","Type":"ContainerDied","Data":"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c"} Nov 03 19:33:40.061996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.061802 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk" event={"ID":"4f0a983e-24ee-4862-bd44-796db6c6079b","Type":"ContainerDied","Data":"b14e2fb88d40fbe49540da30b9a44ab4ab23edc33e14f95a4643b7a12aa635dd"} Nov 03 19:33:40.061996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.061833 5209 scope.go:122] "RemoveContainer" containerID="37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c" Nov 03 19:33:40.063265 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.063242 5209 generic.go:356] "Generic (PLEG): container finished" podID="426d5439-a51f-41c6-99c5-5f914803205f" containerID="64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17" exitCode=0 Nov 03 19:33:40.063391 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.063300 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" Nov 03 19:33:40.063391 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.063307 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" event={"ID":"426d5439-a51f-41c6-99c5-5f914803205f","Type":"ContainerDied","Data":"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17"} Nov 03 19:33:40.063391 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.063350 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf" event={"ID":"426d5439-a51f-41c6-99c5-5f914803205f","Type":"ContainerDied","Data":"fdf69e07949a93a1af0032a78200555da352b50a7eb3685b6695ef82b2438a50"} Nov 03 19:33:40.074412 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.074392 5209 scope.go:122] "RemoveContainer" containerID="37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c" Nov 03 19:33:40.074780 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:33:40.074752 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c\": container with ID starting with 37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c not found: ID does not exist" containerID="37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c" Nov 03 19:33:40.074857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.074793 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c"} err="failed to get container status \"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c\": rpc error: code = NotFound desc = could not find container \"37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c\": container with ID starting with 37aee5bbf529e939b24ccde9a74e6d823e43704c1a9b770412461e82e3c4eb6c not found: ID does not exist" Nov 03 19:33:40.074857 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.074817 5209 scope.go:122] "RemoveContainer" containerID="64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17" Nov 03 19:33:40.083182 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.083164 5209 scope.go:122] "RemoveContainer" containerID="64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17" Nov 03 19:33:40.083439 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:33:40.083421 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17\": container with ID starting with 64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17 not found: ID does not exist" containerID="64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17" Nov 03 19:33:40.083489 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.083446 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17"} err="failed to get container status \"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17\": rpc error: code = NotFound desc = could not find container \"64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17\": container with ID starting with 64ac2f1c998d4d076ae1bd630d0df247307501b656c499a2196166952c9acd17 not found: ID does not exist" Nov 03 19:33:40.215448 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.215405 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:33:40.218992 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.218963 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-826a68-7fc7bc865f-x5xjf"] Nov 03 19:33:40.321264 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.321192 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:33:40.325244 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.325221 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-826a68-8646f89b5d-fjvtk"] Nov 03 19:33:40.697546 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.697514 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426d5439-a51f-41c6-99c5-5f914803205f" path="/var/lib/kubelet/pods/426d5439-a51f-41c6-99c5-5f914803205f/volumes" Nov 03 19:33:40.697870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:40.697857 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0a983e-24ee-4862-bd44-796db6c6079b" path="/var/lib/kubelet/pods/4f0a983e-24ee-4862-bd44-796db6c6079b/volumes" Nov 03 19:33:55.527440 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:33:55.527397 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:34:01.938697 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:01.938659 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:34:03.583988 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.583954 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:34:03.584398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.584169 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" podUID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" containerName="maas-api" containerID="cri-o://e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8" gracePeriod=30 Nov 03 19:34:03.838830 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.838722 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:34:03.888664 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.888626 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") pod \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " Nov 03 19:34:03.888874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.888776 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\" (UniqueName: \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\") pod \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " Nov 03 19:34:03.888874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.888849 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\") pod \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " Nov 03 19:34:03.889008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.888905 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\") pod \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\" (UID: \"1cde17f5-2789-4f0c-a8b4-6377a72ab877\") " Nov 03 19:34:03.889289 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.889256 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca" pod "1cde17f5-2789-4f0c-a8b4-6377a72ab877" (UID: "1cde17f5-2789-4f0c-a8b4-6377a72ab877"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:03.894719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.891443 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls" pod "1cde17f5-2789-4f0c-a8b4-6377a72ab877" (UID: "1cde17f5-2789-4f0c-a8b4-6377a72ab877"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:03.894719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.891474 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx" pod "1cde17f5-2789-4f0c-a8b4-6377a72ab877" (UID: "1cde17f5-2789-4f0c-a8b4-6377a72ab877"). InnerVolumeSpecName "kube-api-access-mnhlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:03.894719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.891560 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs" pod "1cde17f5-2789-4f0c-a8b4-6377a72ab877" (UID: "1cde17f5-2789-4f0c-a8b4-6377a72ab877"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:03.989796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.989755 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhlx\" (UniqueName: \"kubernetes.io/projected/1cde17f5-2789-4f0c-a8b4-6377a72ab877-kube-api-access-mnhlx\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:03.989796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.989788 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cde17f5-2789-4f0c-a8b4-6377a72ab877-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:03.989796 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.989804 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:03.990113 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:03.989817 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cde17f5-2789-4f0c-a8b4-6377a72ab877-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:04.195949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.195863 5209 generic.go:356] "Generic (PLEG): container finished" podID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" containerID="e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8" exitCode=0 Nov 03 19:34:04.195949 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.195909 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" Nov 03 19:34:04.196208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.195945 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" event={"ID":"1cde17f5-2789-4f0c-a8b4-6377a72ab877","Type":"ContainerDied","Data":"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8"} Nov 03 19:34:04.196208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.196000 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl" event={"ID":"1cde17f5-2789-4f0c-a8b4-6377a72ab877","Type":"ContainerDied","Data":"1c68029d93a7408818adbdeac8cba834fe9cb97873c4dd374b290c7c823f3dd5"} Nov 03 19:34:04.196208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.196025 5209 scope.go:122] "RemoveContainer" containerID="e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8" Nov 03 19:34:04.206224 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.206205 5209 scope.go:122] "RemoveContainer" containerID="e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8" Nov 03 19:34:04.206506 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:04.206485 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8\": container with ID starting with e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8 not found: ID does not exist" containerID="e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8" Nov 03 19:34:04.206577 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.206515 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8"} err="failed to get container status \"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8\": rpc error: code = NotFound desc = could not find container \"e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8\": container with ID starting with e12b8524b5a0d218407deedad272eff0560d2f931146195c41e536771bf0f3a8 not found: ID does not exist" Nov 03 19:34:04.349947 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.349903 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:34:04.354208 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.354179 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-826a68-59697f5cd6-lhbpl"] Nov 03 19:34:04.698325 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:04.698289 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" path="/var/lib/kubelet/pods/1cde17f5-2789-4f0c-a8b4-6377a72ab877/volumes" Nov 03 19:34:14.975941 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.975907 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2"] Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976313 5209 cpu_manager.go:414] "containerMap: removing container" podUID="426d5439-a51f-41c6-99c5-5f914803205f" containerName="payload-pre-processing" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976325 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d5439-a51f-41c6-99c5-5f914803205f" containerName="payload-pre-processing" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976338 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976343 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976353 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976358 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976366 5209 cpu_manager.go:414] "containerMap: removing container" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976371 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976377 5209 cpu_manager.go:414] "containerMap: removing container" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976383 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976389 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976393 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976402 5209 cpu_manager.go:414] "containerMap: removing container" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976406 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976412 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976416 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976423 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" containerName="payload-processing" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976427 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" containerName="payload-processing" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976435 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976438 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976445 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976449 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976455 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976459 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.976455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976465 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976470 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976478 5209 cpu_manager.go:414] "containerMap: removing container" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976483 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976489 5209 cpu_manager.go:414] "containerMap: removing container" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976494 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976500 5209 cpu_manager.go:414] "containerMap: removing container" podUID="628cd800-3f40-4244-9818-7dce28abc006" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976505 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628cd800-3f40-4244-9818-7dce28abc006" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976515 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976519 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976529 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976533 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976540 5209 cpu_manager.go:414] "containerMap: removing container" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976544 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976552 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976556 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976561 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976565 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976572 5209 cpu_manager.go:414] "containerMap: removing container" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976576 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976584 5209 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976588 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976595 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976599 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976604 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" containerName="maas-api" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976607 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" containerName="maas-api" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976613 5209 cpu_manager.go:414] "containerMap: removing container" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976617 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976626 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4f0a983e-24ee-4862-bd44-796db6c6079b" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976630 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0a983e-24ee-4862-bd44-796db6c6079b" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976639 5209 cpu_manager.go:414] "containerMap: removing container" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976644 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976652 5209 cpu_manager.go:414] "containerMap: removing container" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976656 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976712 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="af95000b-df6e-4a58-a114-f4e5ca7a5382" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976720 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976726 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976732 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="886455ec-0ba7-4333-b783-3606386adf82" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976738 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a985a-baa9-426b-b5c5-d4d21a64228a" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976743 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976749 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976753 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cde17f5-2789-4f0c-a8b4-6377a72ab877" containerName="maas-api" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976758 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0a983e-24ee-4862-bd44-796db6c6079b" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976764 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="68800080-3f84-4b48-a0ac-31ab9d9b5119" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976769 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976773 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="873a9b12-181c-49ed-9a18-2e27cc380fd3" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976779 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976784 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d5439-a51f-41c6-99c5-5f914803205f" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976789 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976793 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976798 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976803 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e37712-47d5-4bf1-8748-f5eecb0805b7" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976808 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976813 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93f4c25-cc2a-4123-90c7-98eec6b3a0cb" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976817 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f8700c-38c3-4fd6-8d6c-9a879b61c6f5" containerName="payload-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976822 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976826 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab29859-8aec-40ac-ad30-bd0e8e91ab6e" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976830 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f367ac2d-ad2d-4edb-85c5-8e32e7e3cde4" containerName="istio-proxy" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976834 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc7f747-2e30-4cb4-925a-a16ad1167144" containerName="payload-pre-processing" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976840 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976844 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="628cd800-3f40-4244-9818-7dce28abc006" containerName="main" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976848 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:34:14.977228 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.976853 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:34:14.981291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:14.981271 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.043668 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.043638 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2"] Nov 03 19:34:15.087315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.087284 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:34:15.087479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.087365 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:34:15.095215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.095187 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcf2s\" (UniqueName: \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.095302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.095286 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.095355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.095343 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.196204 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.196173 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcf2s\" (UniqueName: \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.196373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.196243 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.196373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.196291 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.196998 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.196974 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.198735 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.198710 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.254378 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.254315 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcf2s\" (UniqueName: \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\") pod \"maas-api-revoke-sub-351cfc0b-m96v2\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.294130 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.294096 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:15.668303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:15.668266 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2"] Nov 03 19:34:15.674003 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:34:15.673968 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc1dbbb_e7ca_45c3_89e8_d867ea203dba.slice/crio-7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8 WatchSource:0}: Error finding container 7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8: Status 404 returned error can't find the container with id 7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8 Nov 03 19:34:16.261997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:16.261913 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" event={"ID":"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba","Type":"ContainerStarted","Data":"446ce38fe73f5688009302ef447d0fc52fb5dbf3e5376e59bfcfb8f8f1f4126c"} Nov 03 19:34:16.261997 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:16.261970 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" event={"ID":"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba","Type":"ContainerStarted","Data":"7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8"} Nov 03 19:34:16.374303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:16.374249 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" podStartSLOduration=2.374233066 podStartE2EDuration="2.374233066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:16.373458056 +0000 UTC m=+2937.784558199" watchObservedRunningTime="2026-11-03 19:34:16.374233066 +0000 UTC m=+2937.785333191" Nov 03 19:34:17.269782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:17.269745 5209 generic.go:356] "Generic (PLEG): container finished" podID="4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" containerID="446ce38fe73f5688009302ef447d0fc52fb5dbf3e5376e59bfcfb8f8f1f4126c" exitCode=0 Nov 03 19:34:17.270196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:17.269811 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" event={"ID":"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba","Type":"ContainerDied","Data":"446ce38fe73f5688009302ef447d0fc52fb5dbf3e5376e59bfcfb8f8f1f4126c"} Nov 03 19:34:18.409039 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.409015 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:18.507460 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.507424 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:34:18.509600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.509574 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:34:18.518017 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.517991 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 19:34:18.527752 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.526894 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\") pod \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " Nov 03 19:34:18.527752 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.527056 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\") pod \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " Nov 03 19:34:18.527752 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.527098 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\" (UniqueName: \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\") pod \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\" (UID: \"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba\") " Nov 03 19:34:18.528694 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.528669 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca" pod "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" (UID: "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:18.536725 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.536693 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s" pod "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" (UID: "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba"). InnerVolumeSpecName "kube-api-access-bcf2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:18.537167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.537150 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca" pod "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" (UID: "4cc1dbbb-e7ca-45c3-89e8-d867ea203dba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:18.628711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.628679 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-xks-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.628711 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.628712 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcf2s\" (UniqueName: \"kubernetes.io/projected/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-kube-api-access-bcf2s\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.628917 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.628724 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba-maas-api-openshift-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.723758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:18.723729 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 19:34:19.281389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:19.281353 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" event={"ID":"4cc1dbbb-e7ca-45c3-89e8-d867ea203dba","Type":"ContainerDied","Data":"7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8"} Nov 03 19:34:19.281575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:19.281404 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae3d0fcbedb6cbbfcc4491232ab241ccc828b63d9cf42c7d31f4a63797a17b8" Nov 03 19:34:19.281575 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:19.281362 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2" Nov 03 19:34:23.724649 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:23.724608 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" containerID="cri-o://ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1" gracePeriod=25 Nov 03 19:34:24.879391 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:24.879355 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:34:31.349559 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:31.349524 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:34:31.936757 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:31.936721 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:34:33.534140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.534058 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" containerID="cri-o://dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6" gracePeriod=45 Nov 03 19:34:33.534518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.534438 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" containerID="cri-o://851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f" gracePeriod=45 Nov 03 19:34:33.549676 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.549642 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" containerID="cri-o://7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d" gracePeriod=45 Nov 03 19:34:33.866761 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.866729 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:34:33.869915 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.869895 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:34:33.873157 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.873140 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:34:33.980816 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.980780 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\" (UniqueName: \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981004 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.980857 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981004 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.980958 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981100 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981010 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981100 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981080 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981115 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\" (UniqueName: \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981161 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981157 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.981360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981220 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981371 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\" (UniqueName: \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981419 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981410 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981374 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.981508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981400 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.981508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981416 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.981508 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981485 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.981645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981510 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\" (UniqueName: \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981542 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981572 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981607 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981645 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981640 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981898 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981679 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\" (UniqueName: \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\") pod \"43d0b444-f833-4f1e-be3a-c7d875ae7799\" (UID: \"43d0b444-f833-4f1e-be3a-c7d875ae7799\") " Nov 03 19:34:33.981898 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981742 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\" (UniqueName: \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\") pod \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\" (UID: \"30838aaa-f1bc-4076-b3da-0b79dbe9d30b\") " Nov 03 19:34:33.981898 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981787 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\") pod \"776454c0-5160-427e-9ae5-cca43cd496da\" (UID: \"776454c0-5160-427e-9ae5-cca43cd496da\") " Nov 03 19:34:33.981898 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.981860 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.982108 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982067 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982235 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982283 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982299 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982312 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982311 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982327 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982341 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982416 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982354 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.982965 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.982557 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.983897 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.983867 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:33.984195 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.984168 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "kube-api-access-f4xk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:33.984562 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.984543 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "kube-api-access-5h2rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:33.985180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.985155 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:33.985180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.985168 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm" pod "43d0b444-f833-4f1e-be3a-c7d875ae7799" (UID: "43d0b444-f833-4f1e-be3a-c7d875ae7799"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.985368 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.985195 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "kube-api-access-lcbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:33.985368 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.985320 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:33.986355 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.986333 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm" pod "776454c0-5160-427e-9ae5-cca43cd496da" (UID: "776454c0-5160-427e-9ae5-cca43cd496da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.986484 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:33.986471 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm" pod "30838aaa-f1bc-4076-b3da-0b79dbe9d30b" (UID: "30838aaa-f1bc-4076-b3da-0b79dbe9d30b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:34.083536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083503 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbr5\" (UniqueName: \"kubernetes.io/projected/776454c0-5160-427e-9ae5-cca43cd496da-kube-api-access-lcbr5\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083530 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776454c0-5160-427e-9ae5-cca43cd496da-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083536 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083540 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083549 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xk5\" (UniqueName: \"kubernetes.io/projected/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-kube-api-access-f4xk5\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083558 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083567 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083575 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d0b444-f833-4f1e-be3a-c7d875ae7799-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083583 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2rr\" (UniqueName: \"kubernetes.io/projected/43d0b444-f833-4f1e-be3a-c7d875ae7799-kube-api-access-5h2rr\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083591 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838aaa-f1bc-4076-b3da-0b79dbe9d30b-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083599 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776454c0-5160-427e-9ae5-cca43cd496da-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.083776 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.083607 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d0b444-f833-4f1e-be3a-c7d875ae7799-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.361461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.361425 5209 generic.go:356] "Generic (PLEG): container finished" podID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerID="7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d" exitCode=0 Nov 03 19:34:34.361680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.361474 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" Nov 03 19:34:34.361680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.361510 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" event={"ID":"43d0b444-f833-4f1e-be3a-c7d875ae7799","Type":"ContainerDied","Data":"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d"} Nov 03 19:34:34.361680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.361553 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6" event={"ID":"43d0b444-f833-4f1e-be3a-c7d875ae7799","Type":"ContainerDied","Data":"aa7572bee213ba4c9293496627fa3c658804117b04cbdc2a9bc29c7ffc1b388e"} Nov 03 19:34:34.361680 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.361570 5209 scope.go:122] "RemoveContainer" containerID="7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d" Nov 03 19:34:34.362872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.362850 5209 generic.go:356] "Generic (PLEG): container finished" podID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerID="dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6" exitCode=0 Nov 03 19:34:34.362963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.362881 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" Nov 03 19:34:34.362963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.362932 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" event={"ID":"30838aaa-f1bc-4076-b3da-0b79dbe9d30b","Type":"ContainerDied","Data":"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6"} Nov 03 19:34:34.363055 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.362968 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn" event={"ID":"30838aaa-f1bc-4076-b3da-0b79dbe9d30b","Type":"ContainerDied","Data":"3c98d241c1b0d19a31f6d86a2fe32dd2ad295eb909b83819f404854c74fd574e"} Nov 03 19:34:34.364527 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.364505 5209 generic.go:356] "Generic (PLEG): container finished" podID="776454c0-5160-427e-9ae5-cca43cd496da" containerID="851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f" exitCode=0 Nov 03 19:34:34.364655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.364562 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" Nov 03 19:34:34.364655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.364565 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" event={"ID":"776454c0-5160-427e-9ae5-cca43cd496da","Type":"ContainerDied","Data":"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f"} Nov 03 19:34:34.364769 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.364667 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj" event={"ID":"776454c0-5160-427e-9ae5-cca43cd496da","Type":"ContainerDied","Data":"aaa42f02eab19c463ca2b1d5d3916802cfe7ecc6dadebdd3343f899d6988b7bd"} Nov 03 19:34:34.373926 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.373904 5209 scope.go:122] "RemoveContainer" containerID="7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d" Nov 03 19:34:34.374310 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:34.374287 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d\": container with ID starting with 7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d not found: ID does not exist" containerID="7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d" Nov 03 19:34:34.374424 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.374317 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d"} err="failed to get container status \"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d\": rpc error: code = NotFound desc = could not find container \"7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d\": container with ID starting with 7c546e05d846142a239af08e8e7589f8be0339edfc5d4c3033ce9fa609a8eb9d not found: ID does not exist" Nov 03 19:34:34.374424 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.374341 5209 scope.go:122] "RemoveContainer" containerID="dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6" Nov 03 19:34:34.383621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.383605 5209 scope.go:122] "RemoveContainer" containerID="dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6" Nov 03 19:34:34.383892 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:34.383875 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6\": container with ID starting with dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6 not found: ID does not exist" containerID="dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6" Nov 03 19:34:34.383943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.383897 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6"} err="failed to get container status \"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6\": rpc error: code = NotFound desc = could not find container \"dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6\": container with ID starting with dd0bfd4f0ba53ace7cf357dd640db0ca85f3bb1c1ea07ed9b48546ccc40921c6 not found: ID does not exist" Nov 03 19:34:34.383943 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.383912 5209 scope.go:122] "RemoveContainer" containerID="851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f" Nov 03 19:34:34.391513 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.391489 5209 scope.go:122] "RemoveContainer" containerID="851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f" Nov 03 19:34:34.391763 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:34.391745 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f\": container with ID starting with 851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f not found: ID does not exist" containerID="851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f" Nov 03 19:34:34.391830 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.391774 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f"} err="failed to get container status \"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f\": rpc error: code = NotFound desc = could not find container \"851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f\": container with ID starting with 851581c293002883beee61c28aec4930a2370fdd2adbf11cc1a61a374c18226f not found: ID does not exist" Nov 03 19:34:34.418961 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.418938 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 19:34:34.419148 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.419114 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" podUID="06329280-b1fd-478d-89c1-a90a4f85a01a" containerName="maas-api" containerID="cri-o://cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd" gracePeriod=30 Nov 03 19:34:34.516846 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.516812 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 19:34:34.529922 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.523859 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/simulator-w3-fa2878-kserve-d6fbf4986-tdkd6"] Nov 03 19:34:34.626940 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.626827 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:34:34.629584 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.629554 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/premium-w3-fa2878-kserve-77c54769f-mw8bj"] Nov 03 19:34:34.690087 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.690063 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 19:34:34.698028 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.698004 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" path="/var/lib/kubelet/pods/43d0b444-f833-4f1e-be3a-c7d875ae7799/volumes" Nov 03 19:34:34.698486 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.698470 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776454c0-5160-427e-9ae5-cca43cd496da" path="/var/lib/kubelet/pods/776454c0-5160-427e-9ae5-cca43cd496da/volumes" Nov 03 19:34:34.732842 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.732816 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:34:34.738723 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.738695 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-2-w3-fa2878-kserve-7b57b9577-ggwgn"] Nov 03 19:34:34.790413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.790382 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\" (UniqueName: \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\") pod \"06329280-b1fd-478d-89c1-a90a4f85a01a\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " Nov 03 19:34:34.790627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.790438 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") pod \"06329280-b1fd-478d-89c1-a90a4f85a01a\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " Nov 03 19:34:34.790627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.790507 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\") pod \"06329280-b1fd-478d-89c1-a90a4f85a01a\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " Nov 03 19:34:34.790627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.790536 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\") pod \"06329280-b1fd-478d-89c1-a90a4f85a01a\" (UID: \"06329280-b1fd-478d-89c1-a90a4f85a01a\") " Nov 03 19:34:34.792038 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.791448 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca" pod "06329280-b1fd-478d-89c1-a90a4f85a01a" (UID: "06329280-b1fd-478d-89c1-a90a4f85a01a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:34.796156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.794857 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls" pod "06329280-b1fd-478d-89c1-a90a4f85a01a" (UID: "06329280-b1fd-478d-89c1-a90a4f85a01a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:34.796156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.795312 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh" pod "06329280-b1fd-478d-89c1-a90a4f85a01a" (UID: "06329280-b1fd-478d-89c1-a90a4f85a01a"). InnerVolumeSpecName "kube-api-access-7kcbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:34.804299 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.804262 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs" pod "06329280-b1fd-478d-89c1-a90a4f85a01a" (UID: "06329280-b1fd-478d-89c1-a90a4f85a01a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:34.891966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.891893 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06329280-b1fd-478d-89c1-a90a4f85a01a-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.891966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.891921 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.891966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.891931 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kcbh\" (UniqueName: \"kubernetes.io/projected/06329280-b1fd-478d-89c1-a90a4f85a01a-kube-api-access-7kcbh\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.891966 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:34.891940 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06329280-b1fd-478d-89c1-a90a4f85a01a-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:35.377292 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.377256 5209 generic.go:356] "Generic (PLEG): container finished" podID="06329280-b1fd-478d-89c1-a90a4f85a01a" containerID="cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd" exitCode=0 Nov 03 19:34:35.377449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.377301 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" Nov 03 19:34:35.377449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.377304 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" event={"ID":"06329280-b1fd-478d-89c1-a90a4f85a01a","Type":"ContainerDied","Data":"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd"} Nov 03 19:34:35.377449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.377425 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2" event={"ID":"06329280-b1fd-478d-89c1-a90a4f85a01a","Type":"ContainerDied","Data":"8177ff796c1923e73c3be80fbca6057ac23b376aad76c387409165fb90155fdf"} Nov 03 19:34:35.377449 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.377449 5209 scope.go:122] "RemoveContainer" containerID="cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd" Nov 03 19:34:35.388006 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.387983 5209 scope.go:122] "RemoveContainer" containerID="cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd" Nov 03 19:34:35.388304 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:35.388286 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd\": container with ID starting with cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd not found: ID does not exist" containerID="cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd" Nov 03 19:34:35.388365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.388312 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd"} err="failed to get container status \"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd\": rpc error: code = NotFound desc = could not find container \"cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd\": container with ID starting with cc5eb4142e4fd80a8e91df6d92fea6e4fda81a21d46e0a6bffe58577f0c125cd not found: ID does not exist" Nov 03 19:34:35.531217 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.531181 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 19:34:35.537364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:35.537334 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-eb6ef8-56f5cd8b58-5xzm2"] Nov 03 19:34:36.698516 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:36.698480 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06329280-b1fd-478d-89c1-a90a4f85a01a" path="/var/lib/kubelet/pods/06329280-b1fd-478d-89c1-a90a4f85a01a/volumes" Nov 03 19:34:36.698909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:36.698872 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" path="/var/lib/kubelet/pods/30838aaa-f1bc-4076-b3da-0b79dbe9d30b/volumes" Nov 03 19:34:41.348811 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:41.348770 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:34:43.970272 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:43.970242 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 19:34:44.079520 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.079493 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\" (UniqueName: \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\") pod \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " Nov 03 19:34:44.079695 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.079535 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\") pod \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\" (UID: \"c33db903-d784-46d8-af5b-a1b1bbff3a0c\") " Nov 03 19:34:44.079953 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.079928 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume" pod "c33db903-d784-46d8-af5b-a1b1bbff3a0c" (UID: "c33db903-d784-46d8-af5b-a1b1bbff3a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:44.081646 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.081620 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm" pod "c33db903-d784-46d8-af5b-a1b1bbff3a0c" (UID: "c33db903-d784-46d8-af5b-a1b1bbff3a0c"). InnerVolumeSpecName "kube-api-access-j29hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:44.180234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.180201 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j29hm\" (UniqueName: \"kubernetes.io/projected/c33db903-d784-46d8-af5b-a1b1bbff3a0c-kube-api-access-j29hm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:44.180234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.180231 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33db903-d784-46d8-af5b-a1b1bbff3a0c-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:44.431927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.431844 5209 generic.go:356] "Generic (PLEG): container finished" podID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerID="ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1" exitCode=0 Nov 03 19:34:44.431927 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.431892 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" Nov 03 19:34:44.432101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.431927 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" event={"ID":"c33db903-d784-46d8-af5b-a1b1bbff3a0c","Type":"ContainerDied","Data":"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1"} Nov 03 19:34:44.432101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.431984 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp" event={"ID":"c33db903-d784-46d8-af5b-a1b1bbff3a0c","Type":"ContainerDied","Data":"a2143978f29088b90d06ca2f90162d1720af7551f01fe60ae02db880ab6bffe2"} Nov 03 19:34:44.432101 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.432012 5209 scope.go:122] "RemoveContainer" containerID="ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1" Nov 03 19:34:44.441330 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.441314 5209 scope.go:122] "RemoveContainer" containerID="ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1" Nov 03 19:34:44.441575 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:34:44.441557 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1\": container with ID starting with ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1 not found: ID does not exist" containerID="ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1" Nov 03 19:34:44.441637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.441587 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1"} err="failed to get container status \"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1\": rpc error: code = NotFound desc = could not find container \"ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1\": container with ID starting with ea657b8ccf6d12c72adc6af12a398e5c8f843c96d684b3498af64c956865cea1 not found: ID does not exist" Nov 03 19:34:44.585056 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.585027 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 19:34:44.631192 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.631155 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-eb6ef8-5db5844c67-r6nhp"] Nov 03 19:34:44.697946 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:44.697857 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" path="/var/lib/kubelet/pods/c33db903-d784-46d8-af5b-a1b1bbff3a0c/volumes" Nov 03 19:34:45.872799 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:45.872770 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:34:59.024238 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:59.024150 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:34:59.024686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:34:59.024406 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" containerID="cri-o://150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19" gracePeriod=30 Nov 03 19:35:01.937515 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:35:01.937474 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:35:03.057583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:03.057545 5209 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:35:03.057985 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:03.057605 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:35:04.277143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.277102 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:35:04.363425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363391 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363425 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363427 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363466 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363596 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363699 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363703 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:04.363809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363761 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363818 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363830 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:04.363967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363873 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363926 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\") pod \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\" (UID: \"95a8049f-43bb-4c92-b308-e52a6ae6e8a2\") " Nov 03 19:35:04.363967 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.363940 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:35:04.364266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.364241 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:04.364266 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.364252 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:04.364386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.364291 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.364386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.364312 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.364386 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.364339 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.365833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.365810 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:04.366004 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.365987 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:35:04.366233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.366213 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:04.366233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.366228 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd" pod "95a8049f-43bb-4c92-b308-e52a6ae6e8a2" (UID: "95a8049f-43bb-4c92-b308-e52a6ae6e8a2"). InnerVolumeSpecName "kube-api-access-8jvpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:04.465438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465406 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvpd\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-kube-api-access-8jvpd\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.465438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465431 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.465438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465442 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.465665 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465452 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.465665 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465461 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.465665 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.465470 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a8049f-43bb-4c92-b308-e52a6ae6e8a2-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:04.552108 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.552074 5209 generic.go:356] "Generic (PLEG): container finished" podID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerID="150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19" exitCode=0 Nov 03 19:35:04.552294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.552146 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" Nov 03 19:35:04.552294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.552155 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" event={"ID":"95a8049f-43bb-4c92-b308-e52a6ae6e8a2","Type":"ContainerDied","Data":"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19"} Nov 03 19:35:04.552294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.552209 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w" event={"ID":"95a8049f-43bb-4c92-b308-e52a6ae6e8a2","Type":"ContainerDied","Data":"071a6f1af8096c3f80e077e9c199c883e7b7b697d8e5499fff79e69becfab392"} Nov 03 19:35:04.552294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.552233 5209 scope.go:122] "RemoveContainer" containerID="150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19" Nov 03 19:35:04.561545 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.561530 5209 scope.go:122] "RemoveContainer" containerID="150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19" Nov 03 19:35:04.561801 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:35:04.561783 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19\": container with ID starting with 150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19 not found: ID does not exist" containerID="150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19" Nov 03 19:35:04.561839 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.561811 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19"} err="failed to get container status \"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19\": rpc error: code = NotFound desc = could not find container \"150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19\": container with ID starting with 150cd8b4543fb834292fbe40e20ee2f49b8054d0ca01ed23dd79af14c0837a19 not found: ID does not exist" Nov 03 19:35:04.706650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.706571 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:35:04.712987 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:04.712954 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-qb84w"] Nov 03 19:35:05.339495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:05.339460 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:35:06.697082 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:06.697049 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" path="/var/lib/kubelet/pods/95a8049f-43bb-4c92-b308-e52a6ae6e8a2/volumes" Nov 03 19:35:20.493720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:35:20.493679 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:35:31.936144 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:35:31.936081 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:36:01.936990 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:36:01.936954 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:36:04.005438 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:36:04.005402 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:36:24.312047 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:36:24.312008 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:36:31.936279 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:36:31.936203 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:36:40.873756 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:36:40.873718 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:36:56.377588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:36:56.377556 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:37:01.936317 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:37:01.936279 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:37:25.103141 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:25.103085 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:37:31.935556 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:37:31.935516 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:37:34.316741 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:34.316707 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:37:34.363255 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:34.363218 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 19:37:41.248195 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:41.248153 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 19:37:46.249215 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:46.249107 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" containerID="cri-o://1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85" gracePeriod=25 Nov 03 19:37:48.954650 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.954617 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq"] Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.954965 5209 cpu_manager.go:414] "containerMap: removing container" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.954976 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.954988 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" containerName="revoke-keys" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.954993 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" containerName="revoke-keys" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955002 5209 cpu_manager.go:414] "containerMap: removing container" podUID="06329280-b1fd-478d-89c1-a90a4f85a01a" containerName="maas-api" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955007 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06329280-b1fd-478d-89c1-a90a4f85a01a" containerName="maas-api" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955016 5209 cpu_manager.go:414] "containerMap: removing container" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" Nov 03 19:37:48.955024 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955021 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955031 5209 cpu_manager.go:414] "containerMap: removing container" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955036 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955045 5209 cpu_manager.go:414] "containerMap: removing container" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955049 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955059 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955064 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955075 5209 cpu_manager.go:414] "containerMap: removing container" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955080 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4638a05d-61eb-4a81-b4dc-f47fd4fb8087" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955090 5209 cpu_manager.go:414] "containerMap: removing container" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955094 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955105 5209 cpu_manager.go:414] "containerMap: removing container" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955109 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955115 5209 cpu_manager.go:414] "containerMap: removing container" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955134 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955206 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca0a289-a7ba-4276-b72a-b0572dbe7195" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955215 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a8049f-43bb-4c92-b308-e52a6ae6e8a2" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955226 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d0b444-f833-4f1e-be3a-c7d875ae7799" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955237 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="5983c826-9d19-4e65-ae77-98484ebde2c7" containerName="cleanup" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955248 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="776454c0-5160-427e-9ae5-cca43cd496da" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955256 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" containerName="revoke-keys" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955265 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="06329280-b1fd-478d-89c1-a90a4f85a01a" containerName="maas-api" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955273 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91bb60f-afed-4f87-a192-283e15d7ad31" containerName="istio-proxy" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955282 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="30838aaa-f1bc-4076-b3da-0b79dbe9d30b" containerName="main" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955290 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33db903-d784-46d8-af5b-a1b1bbff3a0c" containerName="payload-pre-processing" Nov 03 19:37:48.955319 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.955310 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b428bc-a365-4e4b-ac0f-55bd3ceb7ec9" containerName="cleanup" Nov 03 19:37:48.958365 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.958347 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:48.985233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.985200 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95mf\" (UniqueName: \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:48.985367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.985267 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:48.985367 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:48.985324 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.012955 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.012930 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq"] Nov 03 19:37:49.056172 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.056142 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:37:49.056307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.056195 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:37:49.058817 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.058800 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:37:49.058930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.058917 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:37:49.086268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.086243 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95mf\" (UniqueName: \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.086382 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.086286 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.086382 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.086304 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.086870 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.086852 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.088572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.088545 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.143578 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.143552 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95mf\" (UniqueName: \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\") pod \"maas-api-revoke-sub-be8a59ca-k7tfq\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.272169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.272084 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:49.339544 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.339506 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" containerID="cri-o://7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2" gracePeriod=45 Nov 03 19:37:49.383084 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.383050 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" containerID="cri-o://25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf" gracePeriod=45 Nov 03 19:37:49.614978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.614949 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:37:49.642654 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.642629 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq"] Nov 03 19:37:49.648007 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:37:49.647979 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef487997_b8b1_4f34_9e31_9f82b8a3786e.slice/crio-7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba WatchSource:0}: Error finding container 7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba: Status 404 returned error can't find the container with id 7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba Nov 03 19:37:49.649531 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.649514 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:37:49.692226 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692194 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\" (UniqueName: \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.692331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692240 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692270 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692331 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692294 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.692455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692352 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\" (UniqueName: \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692372 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692389 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.692455 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692428 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692485 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692515 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692527 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692554 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\" (UniqueName: \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\") pod \"aacefc91-48a2-4f34-a016-84d2efbd7256\" (UID: \"aacefc91-48a2-4f34-a016-84d2efbd7256\") " Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692581 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692613 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.692657 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692641 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\") pod \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\" (UID: \"bcc1b9ea-a25e-431f-a3d4-7e5f28577346\") " Nov 03 19:37:49.693025 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.692884 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.693089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693039 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.693089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693057 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.693089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693070 5209 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-tmp-dir\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.693089 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693083 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.693342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693311 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.693442 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.693409 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.695043 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695023 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "kube-api-access-282jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:49.695156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695049 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.695156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695080 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:49.695156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695104 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:49.695156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695148 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp" pod "aacefc91-48a2-4f34-a016-84d2efbd7256" (UID: "aacefc91-48a2-4f34-a016-84d2efbd7256"). InnerVolumeSpecName "kube-api-access-qjstp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:49.695303 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.695188 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs" pod "bcc1b9ea-a25e-431f-a3d4-7e5f28577346" (UID: "bcc1b9ea-a25e-431f-a3d4-7e5f28577346"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:49.793852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793824 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjstp\" (UniqueName: \"kubernetes.io/projected/aacefc91-48a2-4f34-a016-84d2efbd7256-kube-api-access-qjstp\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.793852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793848 5209 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-home\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.793852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793858 5209 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-model-cache\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.794067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793868 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.794067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793877 5209 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacefc91-48a2-4f34-a016-84d2efbd7256-dshm\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.794067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793884 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacefc91-48a2-4f34-a016-84d2efbd7256-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.794067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793893 5209 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-tls-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:49.794067 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:49.793902 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282jc\" (UniqueName: \"kubernetes.io/projected/bcc1b9ea-a25e-431f-a3d4-7e5f28577346-kube-api-access-282jc\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.400423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.400380 5209 generic.go:356] "Generic (PLEG): container finished" podID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerID="25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf" exitCode=0 Nov 03 19:37:50.400934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.400445 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" Nov 03 19:37:50.400934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.400468 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" event={"ID":"aacefc91-48a2-4f34-a016-84d2efbd7256","Type":"ContainerDied","Data":"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf"} Nov 03 19:37:50.400934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.400519 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt" event={"ID":"aacefc91-48a2-4f34-a016-84d2efbd7256","Type":"ContainerDied","Data":"c9fb70c29b2c54042b36d9e959249c2748c3160d7d4c1ed2c6373663040399fe"} Nov 03 19:37:50.400934 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.400544 5209 scope.go:122] "RemoveContainer" containerID="25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf" Nov 03 19:37:50.401911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.401824 5209 generic.go:356] "Generic (PLEG): container finished" podID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerID="7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2" exitCode=0 Nov 03 19:37:50.401911 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.401900 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" Nov 03 19:37:50.402003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.401920 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" event={"ID":"bcc1b9ea-a25e-431f-a3d4-7e5f28577346","Type":"ContainerDied","Data":"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2"} Nov 03 19:37:50.402003 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.401965 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw" event={"ID":"bcc1b9ea-a25e-431f-a3d4-7e5f28577346","Type":"ContainerDied","Data":"3707ea38f691e15892af328ee58e8e7ad55cce6257113f9324a76e373d6a6cac"} Nov 03 19:37:50.411364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.411334 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" event={"ID":"ef487997-b8b1-4f34-9e31-9f82b8a3786e","Type":"ContainerStarted","Data":"50212531ee6178e9f61f9fce6f2b88f0385f055e410bcb0cc0380204e0c81fb4"} Nov 03 19:37:50.411507 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.411433 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" event={"ID":"ef487997-b8b1-4f34-9e31-9f82b8a3786e","Type":"ContainerStarted","Data":"7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba"} Nov 03 19:37:50.414334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.414294 5209 scope.go:122] "RemoveContainer" containerID="25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf" Nov 03 19:37:50.414583 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:37:50.414564 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf\": container with ID starting with 25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf not found: ID does not exist" containerID="25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf" Nov 03 19:37:50.414662 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.414590 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf"} err="failed to get container status \"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf\": rpc error: code = NotFound desc = could not find container \"25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf\": container with ID starting with 25f4ac14dc516a5fa180bbaea0a96c63f95626a75cdfbdeaf48fa761c1d196cf not found: ID does not exist" Nov 03 19:37:50.414662 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.414607 5209 scope.go:122] "RemoveContainer" containerID="7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2" Nov 03 19:37:50.422849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.422830 5209 scope.go:122] "RemoveContainer" containerID="7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2" Nov 03 19:37:50.423103 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:37:50.423084 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2\": container with ID starting with 7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2 not found: ID does not exist" containerID="7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2" Nov 03 19:37:50.423177 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.423109 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2"} err="failed to get container status \"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2\": rpc error: code = NotFound desc = could not find container \"7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2\": container with ID starting with 7efc9778e7ad64df78b50ab1a1aa4b60edcaa2d0e3f521e863eb1971fb5f24f2 not found: ID does not exist" Nov 03 19:37:50.553381 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.553351 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:37:50.557621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.557589 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-2-w2-619731-kserve-56b9889d5c-jgwmw"] Nov 03 19:37:50.663635 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.663534 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" podStartSLOduration=2.663520684 podStartE2EDuration="2.663520684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:50.663302285 +0000 UTC m=+3152.074402402" watchObservedRunningTime="2026-11-03 19:37:50.663520684 +0000 UTC m=+3152.074620806" Nov 03 19:37:50.697991 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.697960 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" path="/var/lib/kubelet/pods/bcc1b9ea-a25e-431f-a3d4-7e5f28577346/volumes" Nov 03 19:37:50.767317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.767283 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 19:37:50.771197 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:50.771167 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-619731/simulator-w2-619731-kserve-776559cb6-78prt"] Nov 03 19:37:51.417561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:51.417526 5209 generic.go:356] "Generic (PLEG): container finished" podID="ef487997-b8b1-4f34-9e31-9f82b8a3786e" containerID="50212531ee6178e9f61f9fce6f2b88f0385f055e410bcb0cc0380204e0c81fb4" exitCode=0 Nov 03 19:37:51.417996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:51.417609 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" event={"ID":"ef487997-b8b1-4f34-9e31-9f82b8a3786e","Type":"ContainerDied","Data":"50212531ee6178e9f61f9fce6f2b88f0385f055e410bcb0cc0380204e0c81fb4"} Nov 03 19:37:52.548464 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.548441 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:52.619479 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.619448 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\") pod \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " Nov 03 19:37:52.619661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.619501 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\" (UniqueName: \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\") pod \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " Nov 03 19:37:52.619661 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.619570 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\") pod \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\" (UID: \"ef487997-b8b1-4f34-9e31-9f82b8a3786e\") " Nov 03 19:37:52.619930 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.619906 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca" pod "ef487997-b8b1-4f34-9e31-9f82b8a3786e" (UID: "ef487997-b8b1-4f34-9e31-9f82b8a3786e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:52.621532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.621511 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca" pod "ef487997-b8b1-4f34-9e31-9f82b8a3786e" (UID: "ef487997-b8b1-4f34-9e31-9f82b8a3786e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:52.621637 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.621525 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf" pod "ef487997-b8b1-4f34-9e31-9f82b8a3786e" (UID: "ef487997-b8b1-4f34-9e31-9f82b8a3786e"). InnerVolumeSpecName "kube-api-access-l95mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:52.697420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.697349 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" path="/var/lib/kubelet/pods/aacefc91-48a2-4f34-a016-84d2efbd7256/volumes" Nov 03 19:37:52.720481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.720451 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-xks-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:52.720481 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.720476 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95mf\" (UniqueName: \"kubernetes.io/projected/ef487997-b8b1-4f34-9e31-9f82b8a3786e-kube-api-access-l95mf\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:52.720640 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:52.720488 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef487997-b8b1-4f34-9e31-9f82b8a3786e-maas-api-openshift-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:53.303993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:53.303952 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:37:53.430054 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:53.430018 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" event={"ID":"ef487997-b8b1-4f34-9e31-9f82b8a3786e","Type":"ContainerDied","Data":"7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba"} Nov 03 19:37:53.430054 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:53.430056 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq" Nov 03 19:37:53.430316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:37:53.430063 5209 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f287c915584bcc1cc9ef2f408bb85c52ff188f8e2157c6d7bf6692f75013aba" Nov 03 19:38:01.936624 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:38:01.936533 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:38:03.304518 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:03.304477 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:38:04.298247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:04.298211 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:38:06.492979 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.492949 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 19:38:06.496726 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.496696 5209 generic.go:356] "Generic (PLEG): container finished" podID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerID="1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85" exitCode=0 Nov 03 19:38:06.496850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.496746 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" Nov 03 19:38:06.496850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.496789 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" event={"ID":"f2e6185b-2066-4b1f-9259-84ff64e9e28c","Type":"ContainerDied","Data":"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85"} Nov 03 19:38:06.496850 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.496843 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n" event={"ID":"f2e6185b-2066-4b1f-9259-84ff64e9e28c","Type":"ContainerDied","Data":"56f4db8b839864a2e3e870472c8695c089a9c9c4689f586d22cd3ebb1c55a505"} Nov 03 19:38:06.496963 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.496871 5209 scope.go:122] "RemoveContainer" containerID="1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85" Nov 03 19:38:06.505354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.505340 5209 scope.go:122] "RemoveContainer" containerID="1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85" Nov 03 19:38:06.505615 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:38:06.505595 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85\": container with ID starting with 1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85 not found: ID does not exist" containerID="1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85" Nov 03 19:38:06.505678 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.505626 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85"} err="failed to get container status \"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85\": rpc error: code = NotFound desc = could not find container \"1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85\": container with ID starting with 1dbd33fb4395d369c29e474c360111703e91fc8f5db7be563173cdf3f932ff85 not found: ID does not exist" Nov 03 19:38:06.652932 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.652904 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\") pod \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " Nov 03 19:38:06.653147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.653007 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\" (UniqueName: \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\") pod \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\" (UID: \"f2e6185b-2066-4b1f-9259-84ff64e9e28c\") " Nov 03 19:38:06.653309 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.653286 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume" pod "f2e6185b-2066-4b1f-9259-84ff64e9e28c" (UID: "f2e6185b-2066-4b1f-9259-84ff64e9e28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:38:06.655041 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.655020 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc" pod "f2e6185b-2066-4b1f-9259-84ff64e9e28c" (UID: "f2e6185b-2066-4b1f-9259-84ff64e9e28c"). InnerVolumeSpecName "kube-api-access-c6vzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:06.753771 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.753744 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6vzc\" (UniqueName: \"kubernetes.io/projected/f2e6185b-2066-4b1f-9259-84ff64e9e28c-kube-api-access-c6vzc\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:06.753771 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.753769 5209 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e6185b-2066-4b1f-9259-84ff64e9e28c-plugins-config-volume\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:06.904886 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.904838 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 19:38:06.907960 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:06.907930 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-619731-7cb96fdb68-f6z8n"] Nov 03 19:38:08.699173 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:08.699142 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" path="/var/lib/kubelet/pods/f2e6185b-2066-4b1f-9259-84ff64e9e28c/volumes" Nov 03 19:38:19.570607 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:19.570575 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:38:29.066196 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:29.066155 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:38:29.066781 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:29.066413 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" containerID="cri-o://099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667" gracePeriod=30 Nov 03 19:38:31.936242 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:38:31.936206 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:38:32.915404 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:32.915360 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kmlnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:38:32.915612 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:32.915418 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:38:34.313883 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.313859 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:38:34.393029 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.392960 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393180 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393051 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393231 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393174 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393229 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393268 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393256 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393290 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393327 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393367 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393461 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393402 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\") pod \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\" (UID: \"22b3c85b-38db-4bcf-8ce3-2a55989e8c04\") " Nov 03 19:38:34.393571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393470 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:38:34.393571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393510 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:38:34.393682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393560 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:38:34.393727 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393708 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:38:34.393767 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393728 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:38:34.393812 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393795 5209 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-data\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.393879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393814 5209 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.393879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393829 5209 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-credential-socket\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.393879 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.393843 5209 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-workload-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.395364 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.395330 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:34.395535 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.395510 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:38:34.395621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.395548 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:38:34.395621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.395577 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn" pod "22b3c85b-38db-4bcf-8ce3-2a55989e8c04" (UID: "22b3c85b-38db-4bcf-8ce3-2a55989e8c04"). InnerVolumeSpecName "kube-api-access-dmrnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:34.494672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.494636 5209 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-envoy\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.494672 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.494676 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrnn\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-kube-api-access-dmrnn\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.494925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.494696 5209 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istiod-ca-cert\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.494925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.494711 5209 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-podinfo\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.494925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.494725 5209 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b3c85b-38db-4bcf-8ce3-2a55989e8c04-istio-token\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:34.649417 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.649321 5209 generic.go:356] "Generic (PLEG): container finished" podID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerID="099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667" exitCode=0 Nov 03 19:38:34.649417 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.649370 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" Nov 03 19:38:34.649631 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.649416 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" event={"ID":"22b3c85b-38db-4bcf-8ce3-2a55989e8c04","Type":"ContainerDied","Data":"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667"} Nov 03 19:38:34.649631 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.649471 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh" event={"ID":"22b3c85b-38db-4bcf-8ce3-2a55989e8c04","Type":"ContainerDied","Data":"fc357adf6197ce86fd79be10397836ca931e744c97958d16ac8c38e564bbe9c6"} Nov 03 19:38:34.649631 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.649496 5209 scope.go:122] "RemoveContainer" containerID="099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667" Nov 03 19:38:34.658802 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.658787 5209 scope.go:122] "RemoveContainer" containerID="099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667" Nov 03 19:38:34.659080 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:38:34.659062 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667\": container with ID starting with 099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667 not found: ID does not exist" containerID="099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667" Nov 03 19:38:34.659137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.659088 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667"} err="failed to get container status \"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667\": rpc error: code = NotFound desc = could not find container \"099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667\": container with ID starting with 099dc621e5675f1b55a2abad90bfd92838cd645b2e797304a5293be07e282667 not found: ID does not exist" Nov 03 19:38:34.803163 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.803130 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:38:34.807057 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:34.807035 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kmlnh"] Nov 03 19:38:36.697350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:38:36.697316 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" path="/var/lib/kubelet/pods/22b3c85b-38db-4bcf-8ce3-2a55989e8c04/volumes" Nov 03 19:39:01.936269 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:39:01.936231 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:39:19.058343 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:19.058305 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2"] Nov 03 19:39:19.103734 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:19.103698 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351cfc0b-m96v2"] Nov 03 19:39:20.697152 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:20.697106 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc1dbbb-e7ca-45c3-89e8-d867ea203dba" path="/var/lib/kubelet/pods/4cc1dbbb-e7ca-45c3-89e8-d867ea203dba/volumes" Nov 03 19:39:31.000829 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:31.000748 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:39:31.940249 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:39:31.940207 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:39:44.641951 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.641915 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65"] Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642291 5209 cpu_manager.go:414] "containerMap: removing container" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642303 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642315 5209 cpu_manager.go:414] "containerMap: removing container" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642322 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642332 5209 cpu_manager.go:414] "containerMap: removing container" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642337 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642349 5209 cpu_manager.go:414] "containerMap: removing container" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642354 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642380 5209 cpu_manager.go:414] "containerMap: removing container" podUID="ef487997-b8b1-4f34-9e31-9f82b8a3786e" containerName="revoke-keys" Nov 03 19:39:44.642444 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642387 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef487997-b8b1-4f34-9e31-9f82b8a3786e" containerName="revoke-keys" Nov 03 19:39:44.642753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642459 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef487997-b8b1-4f34-9e31-9f82b8a3786e" containerName="revoke-keys" Nov 03 19:39:44.642753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642470 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e6185b-2066-4b1f-9259-84ff64e9e28c" containerName="payload-processing" Nov 03 19:39:44.642753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642478 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b3c85b-38db-4bcf-8ce3-2a55989e8c04" containerName="istio-proxy" Nov 03 19:39:44.642753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642484 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacefc91-48a2-4f34-a016-84d2efbd7256" containerName="main" Nov 03 19:39:44.642753 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.642489 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc1b9ea-a25e-431f-a3d4-7e5f28577346" containerName="main" Nov 03 19:39:44.646874 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.646855 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716355 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716397 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716431 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716506 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716550 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716526 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716556 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716609 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg9h\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-kube-api-access-vzg9h\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716634 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.716738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.716707 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.745593 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.745563 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65"] Nov 03 19:39:44.818086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818049 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818086 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818090 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818113 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818157 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818190 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818214 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg9h\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-kube-api-access-vzg9h\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818240 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818358 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818302 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818362 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818643 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818563 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818666 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.818785 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.818690 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.819184 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.819154 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.819324 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.819153 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.820533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.820512 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.820692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.820678 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.879143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.879093 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.879242 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.879207 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg9h\" (UniqueName: \"kubernetes.io/projected/8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4-kube-api-access-vzg9h\") pod \"maas-default-gateway-openshift-default-77f556d647-ftr65\" (UID: \"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:44.959309 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:44.959243 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:45.383809 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:45.383787 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65"] Nov 03 19:39:45.385618 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:39:45.385594 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e09e9d6_6ee8_4119_99d1_4da7dde7e8c4.slice/crio-0ff839456a1d735f0ec4d0a793cc40fc659bef146b7701ec501797c136de5d50 WatchSource:0}: Error finding container 0ff839456a1d735f0ec4d0a793cc40fc659bef146b7701ec501797c136de5d50: Status 404 returned error can't find the container with id 0ff839456a1d735f0ec4d0a793cc40fc659bef146b7701ec501797c136de5d50 Nov 03 19:39:45.387574 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:45.387542 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:45.387656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:45.387601 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:45.387656 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:45.387632 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:46.004064 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.004027 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" event={"ID":"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4","Type":"ContainerStarted","Data":"62fded1ad4cf97011186c655aa375d68d45993abc196c1292819ff4114926e4c"} Nov 03 19:39:46.004506 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.004092 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" event={"ID":"8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4","Type":"ContainerStarted","Data":"0ff839456a1d735f0ec4d0a793cc40fc659bef146b7701ec501797c136de5d50"} Nov 03 19:39:46.125467 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.125415 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podStartSLOduration=2.125400874 podStartE2EDuration="2.125400874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:46.124626693 +0000 UTC m=+3267.535726836" watchObservedRunningTime="2026-11-03 19:39:46.125400874 +0000 UTC m=+3267.536500997" Nov 03 19:39:46.959701 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.959671 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:46.961159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.961136 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:46.961274 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:46.961186 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:47.959872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:47.959833 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:47.960397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:47.959908 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:48.959883 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:48.959846 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:48.960314 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:48.959902 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:49.960187 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:49.960147 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:49.960614 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:49.960208 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:50.960360 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:50.960318 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:50.960755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:50.960379 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:51.959837 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:51.959805 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:39:51.960013 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:51.959855 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 19:39:52.812682 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:52.812642 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:39:53.961397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:53.961350 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:39:53.961884 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:53.961411 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:39:55.003302 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:55.003262 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ftr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:39:55.003734 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:55.003449 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" podUID="8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:39:55.592777 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:55.592746 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:55.593243 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:55.593206 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:55.593942 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:55.593923 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ftr65" Nov 03 19:39:59.776973 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.776928 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85"] Nov 03 19:39:59.783103 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.783077 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855402 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855359 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855402 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855405 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855436 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855546 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855579 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjgm\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-kube-api-access-nvjgm\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855655 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855921 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855706 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855921 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855731 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.855921 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.855808 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.889316 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.889280 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85"] Nov 03 19:39:59.956849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.956797 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.956849 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.956851 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.956891 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.956934 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.956980 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.957010 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.957058 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjgm\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-kube-api-access-nvjgm\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957175 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.957133 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957545 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.957200 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.957803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.957768 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.958034 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.958006 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.958185 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.958162 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.958411 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.958381 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.958530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.958505 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.960429 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.960400 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:39:59.960581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:39:59.960554 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:00.019996 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.019953 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:00.020206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.020106 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjgm\" (UniqueName: \"kubernetes.io/projected/679d0f01-dcb1-48f6-bcd4-cf627ecd8e95-kube-api-access-nvjgm\") pod \"maas-default-gateway-openshift-default-77f556d647-dxk85\" (UID: \"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:00.099872 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.099839 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:00.526251 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.526225 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85"] Nov 03 19:40:00.528631 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:40:00.528605 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679d0f01_dcb1_48f6_bcd4_cf627ecd8e95.slice/crio-a0f27080ce7462a24c90bc499bd105f1c03d34b713a7dd7023e93e87b441b39b WatchSource:0}: Error finding container a0f27080ce7462a24c90bc499bd105f1c03d34b713a7dd7023e93e87b441b39b: Status 404 returned error can't find the container with id a0f27080ce7462a24c90bc499bd105f1c03d34b713a7dd7023e93e87b441b39b Nov 03 19:40:00.530889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.530856 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:00.530994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.530921 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:00.530994 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:00.530951 5209 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:01.107186 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:01.107150 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" event={"ID":"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95","Type":"ContainerStarted","Data":"f3a4da978fde1b2b6a839b2e937be6d3fd0efb2a2bee2a6e0dd405f71b93104b"} Nov 03 19:40:01.107600 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:01.107208 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" event={"ID":"679d0f01-dcb1-48f6-bcd4-cf627ecd8e95","Type":"ContainerStarted","Data":"a0f27080ce7462a24c90bc499bd105f1c03d34b713a7dd7023e93e87b441b39b"} Nov 03 19:40:01.228876 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:01.228813 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podStartSLOduration=2.228799806 podStartE2EDuration="2.228799806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:01.22779761 +0000 UTC m=+3282.638897753" watchObservedRunningTime="2026-11-03 19:40:01.228799806 +0000 UTC m=+3282.639899928" Nov 03 19:40:01.935843 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:40:01.935804 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:40:02.101397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:02.101345 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:02.102990 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:02.102963 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:02.103132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:02.103011 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:02.385156 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:02.384333 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:40:03.101477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:03.101442 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:03.101655 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:03.101500 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:04.101495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:04.101454 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:04.101918 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:04.101523 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:05.101144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:05.100532 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:05.101144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:05.100600 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:06.101486 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:06.101449 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:06.101907 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:06.101503 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:07.100300 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:07.100265 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:40:07.100487 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:07.100320 5209 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 19:40:08.779871 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:08.779841 5209 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:08.780361 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:08.780146 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:08.781139 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:08.781098 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" Nov 03 19:40:22.861137 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:22.861086 5209 scope.go:122] "RemoveContainer" containerID="446ce38fe73f5688009302ef447d0fc52fb5dbf3e5376e59bfcfb8f8f1f4126c" Nov 03 19:40:24.208397 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:24.208361 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:40:31.941826 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:40:31.941773 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:40:42.502353 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:40:42.502319 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:41:01.936886 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:41:01.936798 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:41:12.024715 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:41:12.024681 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:41:22.803833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:41:22.803799 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:41:31.973581 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:41:31.973545 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:41:35.319579 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:41:35.319546 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:41:58.095159 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:41:58.095113 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:42:01.936109 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:42:01.936070 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:42:06.026430 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:06.026388 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:42:26.394206 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:26.394171 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:42:31.936210 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:42:31.936110 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:42:38.927076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:38.927045 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:42:53.045171 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:53.045115 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq"] Nov 03 19:42:53.091259 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:53.091223 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8a59ca-k7tfq"] Nov 03 19:42:54.697571 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:54.697531 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef487997-b8b1-4f34-9e31-9f82b8a3786e" path="/var/lib/kubelet/pods/ef487997-b8b1-4f34-9e31-9f82b8a3786e/volumes" Nov 03 19:42:58.531512 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:42:58.531475 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:43:01.936552 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:43:01.936517 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:43:17.333675 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:43:17.333637 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:43:31.936482 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:43:31.936437 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:44:01.936503 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:44:01.936420 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:44:22.903459 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:44:22.903423 5209 scope.go:122] "RemoveContainer" containerID="50212531ee6178e9f61f9fce6f2b88f0385f055e410bcb0cc0380204e0c81fb4" Nov 03 19:44:30.813673 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:44:30.813639 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:44:31.936726 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:44:31.936682 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:44:50.316305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:44:50.316271 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:45:00.165845 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.165808 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:00.169312 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.169291 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.217790 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.217751 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:45:00.264422 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.264389 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:45:00.264580 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.264455 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:45:00.266290 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.266269 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:00.340925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.340889 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.340925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.340927 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsnh\" (UniqueName: \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.442329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.442253 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.442329 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.442289 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsnh\" (UniqueName: \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.442884 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.442866 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.499621 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.499590 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsnh\" (UniqueName: \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\") pod \"maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:00.782581 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:00.782498 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:01.148738 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:01.148704 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:01.155301 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:45:01.155272 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0613d1_bdc9_44a9_b3da_33906df0fc39.slice/crio-8a011ee376f8fce3ba32497f635f42db2a4f13733dd15721a5590ef3aa39e1d9 WatchSource:0}: Error finding container 8a011ee376f8fce3ba32497f635f42db2a4f13733dd15721a5590ef3aa39e1d9: Status 404 returned error can't find the container with id 8a011ee376f8fce3ba32497f635f42db2a4f13733dd15721a5590ef3aa39e1d9 Nov 03 19:45:01.749334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:01.749237 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerStarted","Data":"ec35c32941ee385ddc309c6bfdaa1efc6caacc1270c52b705f40127d58ebb347"} Nov 03 19:45:01.749334 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:01.749302 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerStarted","Data":"8a011ee376f8fce3ba32497f635f42db2a4f13733dd15721a5590ef3aa39e1d9"} Nov 03 19:45:01.858383 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:01.858322 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" podStartSLOduration=1.858308236 podStartE2EDuration="1.858308236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.856516727 +0000 UTC m=+3583.267616852" watchObservedRunningTime="2026-11-03 19:45:01.858308236 +0000 UTC m=+3583.269408359" Nov 03 19:45:01.936693 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:45:01.936651 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:45:02.756209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:02.756176 5209 generic.go:356] "Generic (PLEG): container finished" podID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerID="ec35c32941ee385ddc309c6bfdaa1efc6caacc1270c52b705f40127d58ebb347" exitCode=60 Nov 03 19:45:02.756740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:02.756276 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerDied","Data":"ec35c32941ee385ddc309c6bfdaa1efc6caacc1270c52b705f40127d58ebb347"} Nov 03 19:45:02.756740 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:02.756543 5209 scope.go:122] "RemoveContainer" containerID="ec35c32941ee385ddc309c6bfdaa1efc6caacc1270c52b705f40127d58ebb347" Nov 03 19:45:03.762881 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:03.762843 5209 generic.go:356] "Generic (PLEG): container finished" podID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerID="e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f" exitCode=60 Nov 03 19:45:03.763333 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:03.762930 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerDied","Data":"e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f"} Nov 03 19:45:03.763333 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:03.762995 5209 scope.go:122] "RemoveContainer" containerID="ec35c32941ee385ddc309c6bfdaa1efc6caacc1270c52b705f40127d58ebb347" Nov 03 19:45:03.763333 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:03.763299 5209 scope.go:122] "RemoveContainer" containerID="e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f" Nov 03 19:45:03.763602 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:45:03.763578 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6_odh-ai-gateway-infra(1e0613d1-bdc9-44a9-b3da-33906df0fc39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" Nov 03 19:45:04.769687 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:04.769658 5209 scope.go:122] "RemoveContainer" containerID="e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f" Nov 03 19:45:04.770096 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:45:04.769891 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6_odh-ai-gateway-infra(1e0613d1-bdc9-44a9-b3da-33906df0fc39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" Nov 03 19:45:13.921236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:13.921198 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:45:18.695667 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:18.695627 5209 scope.go:122] "RemoveContainer" containerID="e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f" Nov 03 19:45:19.841703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:19.841668 5209 generic.go:356] "Generic (PLEG): container finished" podID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerID="12d8e9c75023776b6d3582a4d1b92446c1622c82892d3059ddaea7f1c387ded9" exitCode=60 Nov 03 19:45:19.842153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:19.841741 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerDied","Data":"12d8e9c75023776b6d3582a4d1b92446c1622c82892d3059ddaea7f1c387ded9"} Nov 03 19:45:19.842153 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:19.841832 5209 scope.go:122] "RemoveContainer" containerID="e8f65a6865fbf787e63f19de90384ac6bb988ae1283e8d34ac8d7b1aff1caa4f" Nov 03 19:45:19.842318 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:19.842201 5209 scope.go:122] "RemoveContainer" containerID="12d8e9c75023776b6d3582a4d1b92446c1622c82892d3059ddaea7f1c387ded9" Nov 03 19:45:19.842458 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:45:19.842437 5209 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6_odh-ai-gateway-infra(1e0613d1-bdc9-44a9-b3da-33906df0fc39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" Nov 03 19:45:20.958993 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:20.958956 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:21.094989 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.094965 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:21.230933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.230860 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\" (UniqueName: \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\") pod \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " Nov 03 19:45:21.230933 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.230899 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\") pod \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\" (UID: \"1e0613d1-bdc9-44a9-b3da-33906df0fc39\") " Nov 03 19:45:21.231413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.231385 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca" pod "1e0613d1-bdc9-44a9-b3da-33906df0fc39" (UID: "1e0613d1-bdc9-44a9-b3da-33906df0fc39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:21.233076 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.233051 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh" pod "1e0613d1-bdc9-44a9-b3da-33906df0fc39" (UID: "1e0613d1-bdc9-44a9-b3da-33906df0fc39"). InnerVolumeSpecName "kube-api-access-8wsnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:21.331530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.331494 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsnh\" (UniqueName: \"kubernetes.io/projected/1e0613d1-bdc9-44a9-b3da-33906df0fc39-kube-api-access-8wsnh\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:21.331530 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.331526 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0613d1-bdc9-44a9-b3da-33906df0fc39-maas-api-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:21.853939 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.853905 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" event={"ID":"1e0613d1-bdc9-44a9-b3da-33906df0fc39","Type":"ContainerDied","Data":"8a011ee376f8fce3ba32497f635f42db2a4f13733dd15721a5590ef3aa39e1d9"} Nov 03 19:45:21.854144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.853964 5209 scope.go:122] "RemoveContainer" containerID="12d8e9c75023776b6d3582a4d1b92446c1622c82892d3059ddaea7f1c387ded9" Nov 03 19:45:21.854144 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:21.853922 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6" Nov 03 19:45:22.007728 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:22.007692 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:22.011875 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:22.011850 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-8ecb68-2989558ngpm6"] Nov 03 19:45:22.697008 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:22.696964 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" path="/var/lib/kubelet/pods/1e0613d1-bdc9-44a9-b3da-33906df0fc39/volumes" Nov 03 19:45:31.936408 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:45:31.936314 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:45:34.200675 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:34.200638 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:45:57.387026 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:45:57.386986 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:46:01.948879 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:46:01.948830 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:46:31.936213 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:46:31.936177 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:47:01.936265 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:47:01.936174 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:47:31.936919 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:47:31.936886 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:47:49.705269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:47:49.705231 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:48:01.936591 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:48:01.936548 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:48:07.218020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:48:07.217982 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:48:31.936750 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:48:31.936626 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:48:48.321247 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:48:48.321208 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:49:01.937110 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:49:01.937068 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:49:07.990257 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:49:07.990220 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:49:31.306618 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:49:31.306578 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:49:31.936925 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:49:31.936881 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:49:46.022948 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:49:46.022909 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:50:01.936551 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:50:01.936461 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:50:06.831418 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:50:06.831379 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:50:21.985019 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:50:21.984982 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:50:24.830145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:50:24.827226 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dxk85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:50:24.830145 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:50:24.827296 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dxk85" podUID="679d0f01-dcb1-48f6-bcd4-cf627ecd8e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:31.936397 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:50:31.936360 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:51:01.935806 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:51:01.935771 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:51:03.640787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:03.640754 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:51:30.425275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:30.425186 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:51:31.946725 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:51:31.946689 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:51:33.911294 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:33.911250 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:51:33.911978 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:33.911943 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:45.978236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:45.978200 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:51:48.905143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:48.904275 5209 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wxx42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:51:48.905143 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:51:48.904344 5209 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wxx42" podUID="b5f6b782-1721-4ad3-bf95-63ee93f71a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:01.936190 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:52:01.936154 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:52:01.992703 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:01.992668 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:52:14.416787 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:14.416752 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:52:25.635280 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:25.635243 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:52:31.936360 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:52:31.936318 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:52:48.221342 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.221308 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:52:48.726209 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726175 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:48.726524 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726512 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.726583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726525 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.726583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726538 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.726583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726543 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.726686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726636 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.726686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.726645 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:48.729463 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.729440 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:48.834533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.834500 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:52:48.834533 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.834529 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:52:48.834739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.834626 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zdsbl\"" Nov 03 19:52:48.834739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.834663 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 19:52:48.834739 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.834664 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:52:48.836452 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.836435 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:48.901659 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.901629 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvwg\" (UniqueName: \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:48.901824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.901663 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:48.901824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.901683 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:48.901824 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:48.901810 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.002532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.002457 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.002532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.002492 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.002720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.002553 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.002720 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.002656 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvwg\" (UniqueName: \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.003305 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.003283 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.004935 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.004910 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.005035 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.004978 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.063398 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.063369 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvwg\" (UniqueName: \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\") pod \"maas-api-5c48877c75-99fnv\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.342349 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.342318 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:49.715764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.715740 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:49.718146 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:52:49.718102 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a2317_7e48_452b_92ae_908d2571a215.slice/crio-b0b73b154414ee5e87ba10f44a96e877a187da735ae6bb231feb325c758d43e4 WatchSource:0}: Error finding container b0b73b154414ee5e87ba10f44a96e877a187da735ae6bb231feb325c758d43e4: Status 404 returned error can't find the container with id b0b73b154414ee5e87ba10f44a96e877a187da735ae6bb231feb325c758d43e4 Nov 03 19:52:49.719430 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:49.719411 5209 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:52:50.188373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:50.188337 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" event={"ID":"d32a2317-7e48-452b-92ae-908d2571a215","Type":"ContainerStarted","Data":"b0b73b154414ee5e87ba10f44a96e877a187da735ae6bb231feb325c758d43e4"} Nov 03 19:52:51.195352 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:51.195315 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" event={"ID":"d32a2317-7e48-452b-92ae-908d2571a215","Type":"ContainerStarted","Data":"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef"} Nov 03 19:52:51.195764 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:51.195440 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:51.313585 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:51.313529 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" podStartSLOduration=2.402442918 podStartE2EDuration="3.313516656s" totalImagesPullingTime="911.073738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:51.313249161 +0000 UTC m=+4052.724349324" watchObservedRunningTime="2026-11-03 19:52:51.313516656 +0000 UTC m=+4052.724616775" Nov 03 19:52:57.189844 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.189802 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:57.190307 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.190046 5209 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" podUID="d32a2317-7e48-452b-92ae-908d2571a215" containerName="maas-api" containerID="cri-o://56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef" gracePeriod=30 Nov 03 19:52:57.195636 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.195612 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:57.239866 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.239839 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-vqwqr"] Nov 03 19:52:57.240269 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.240257 5209 cpu_manager.go:414] "containerMap: removing container" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:57.240311 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.240271 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:57.240354 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.240330 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0613d1-bdc9-44a9-b3da-33906df0fc39" containerName="cleanup" Nov 03 19:52:57.243090 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.243066 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.279020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.278995 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gd9\" (UniqueName: \"kubernetes.io/projected/55fb4b93-d664-4763-9002-6b45c716ffbb-kube-api-access-k9gd9\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.279167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.279077 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55fb4b93-d664-4763-9002-6b45c716ffbb-openshift-service-ca\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.279167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.279104 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-metrics-certs\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.279293 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.279272 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-maas-api-tls\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.285420 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.285374 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-vqwqr"] Nov 03 19:52:57.380630 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.380595 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-maas-api-tls\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.380806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.380644 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gd9\" (UniqueName: \"kubernetes.io/projected/55fb4b93-d664-4763-9002-6b45c716ffbb-kube-api-access-k9gd9\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.380806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.380684 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55fb4b93-d664-4763-9002-6b45c716ffbb-openshift-service-ca\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.380806 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.380715 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-metrics-certs\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.381604 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.381577 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55fb4b93-d664-4763-9002-6b45c716ffbb-openshift-service-ca\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.383598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.383573 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-metrics-certs\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.384851 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.384821 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55fb4b93-d664-4763-9002-6b45c716ffbb-maas-api-tls\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.441155 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.441085 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:57.442545 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.442522 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gd9\" (UniqueName: \"kubernetes.io/projected/55fb4b93-d664-4763-9002-6b45c716ffbb-kube-api-access-k9gd9\") pod \"maas-api-595cc648c-vqwqr\" (UID: \"55fb4b93-d664-4763-9002-6b45c716ffbb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.481545 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.481515 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\" (UniqueName: \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\") pod \"d32a2317-7e48-452b-92ae-908d2571a215\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " Nov 03 19:52:57.481686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.481560 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\") pod \"d32a2317-7e48-452b-92ae-908d2571a215\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " Nov 03 19:52:57.481686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.481635 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\") pod \"d32a2317-7e48-452b-92ae-908d2571a215\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " Nov 03 19:52:57.481686 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.481655 5209 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\") pod \"d32a2317-7e48-452b-92ae-908d2571a215\" (UID: \"d32a2317-7e48-452b-92ae-908d2571a215\") " Nov 03 19:52:57.481974 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.481940 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca" pod "d32a2317-7e48-452b-92ae-908d2571a215" (UID: "d32a2317-7e48-452b-92ae-908d2571a215"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:57.483734 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.483704 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg" pod "d32a2317-7e48-452b-92ae-908d2571a215" (UID: "d32a2317-7e48-452b-92ae-908d2571a215"). InnerVolumeSpecName "kube-api-access-vkvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:57.483819 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.483789 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs" pod "d32a2317-7e48-452b-92ae-908d2571a215" (UID: "d32a2317-7e48-452b-92ae-908d2571a215"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:57.483819 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.483805 5209 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls" pod "d32a2317-7e48-452b-92ae-908d2571a215" (UID: "d32a2317-7e48-452b-92ae-908d2571a215"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:57.576644 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.576614 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:57.583082 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.583057 5209 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvwg\" (UniqueName: \"kubernetes.io/projected/d32a2317-7e48-452b-92ae-908d2571a215-kube-api-access-vkvwg\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.583082 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.583081 5209 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-metrics-certs\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.583291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.583092 5209 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2317-7e48-452b-92ae-908d2571a215-openshift-service-ca\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.583291 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.583102 5209 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2317-7e48-452b-92ae-908d2571a215-maas-api-tls\") on node \"ip-10-0-133-236.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.950482 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:57.950460 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-vqwqr"] Nov 03 19:52:57.952903 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:52:57.952865 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fb4b93_d664_4763_9002_6b45c716ffbb.slice/crio-5502e44bb238e2723f363d5a2c632d89eff7200864feb63657e8f327a9c1cdc4 WatchSource:0}: Error finding container 5502e44bb238e2723f363d5a2c632d89eff7200864feb63657e8f327a9c1cdc4: Status 404 returned error can't find the container with id 5502e44bb238e2723f363d5a2c632d89eff7200864feb63657e8f327a9c1cdc4 Nov 03 19:52:58.229750 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.229718 5209 generic.go:356] "Generic (PLEG): container finished" podID="d32a2317-7e48-452b-92ae-908d2571a215" containerID="56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef" exitCode=0 Nov 03 19:52:58.230147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.229772 5209 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" Nov 03 19:52:58.230147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.229800 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" event={"ID":"d32a2317-7e48-452b-92ae-908d2571a215","Type":"ContainerDied","Data":"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef"} Nov 03 19:52:58.230147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.229848 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-99fnv" event={"ID":"d32a2317-7e48-452b-92ae-908d2571a215","Type":"ContainerDied","Data":"b0b73b154414ee5e87ba10f44a96e877a187da735ae6bb231feb325c758d43e4"} Nov 03 19:52:58.230147 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.229869 5209 scope.go:122] "RemoveContainer" containerID="56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef" Nov 03 19:52:58.231167 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.231034 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" event={"ID":"55fb4b93-d664-4763-9002-6b45c716ffbb","Type":"ContainerStarted","Data":"5502e44bb238e2723f363d5a2c632d89eff7200864feb63657e8f327a9c1cdc4"} Nov 03 19:52:58.238975 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.238959 5209 scope.go:122] "RemoveContainer" containerID="56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef" Nov 03 19:52:58.239263 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:52:58.239247 5209 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef\": container with ID starting with 56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef not found: ID does not exist" containerID="56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef" Nov 03 19:52:58.239317 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.239269 5209 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef"} err="failed to get container status \"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef\": rpc error: code = NotFound desc = could not find container \"56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef\": container with ID starting with 56ae58a2d3d5ed736b88be693ce679f37a3fe77342dc0500d908d2713261e5ef not found: ID does not exist" Nov 03 19:52:58.394194 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.394162 5209 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:58.444803 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.444762 5209 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-99fnv"] Nov 03 19:52:58.697465 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:58.697432 5209 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32a2317-7e48-452b-92ae-908d2571a215" path="/var/lib/kubelet/pods/d32a2317-7e48-452b-92ae-908d2571a215/volumes" Nov 03 19:52:59.036254 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:59.036136 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:52:59.237889 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:59.237852 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" event={"ID":"55fb4b93-d664-4763-9002-6b45c716ffbb","Type":"ContainerStarted","Data":"39016491e055c6fbac645a3c1e5dfd4451c560ede2f43a90515cf19fb9ab2159"} Nov 03 19:52:59.238389 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:59.238023 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:52:59.358956 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:52:59.358880 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" podStartSLOduration=1.461699687 podStartE2EDuration="2.358861968s" totalImagesPullingTime="897.162281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:59.355435642 +0000 UTC m=+4060.766535778" watchObservedRunningTime="2026-11-03 19:52:59.358861968 +0000 UTC m=+4060.769962092" Nov 03 19:53:01.943144 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:53:01.942167 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:53:05.250537 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:53:05.250506 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-vqwqr" Nov 03 19:53:10.228692 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:53:10.228658 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:53:31.936517 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:53:31.936477 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:53:33.124844 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:53:33.124802 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:54:01.936414 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:54:01.936377 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:54:04.638058 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:04.638025 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9kbss"] Nov 03 19:54:15.517583 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:15.517550 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xsfxn_8ede7475-dec7-4897-adc6-53c013df7362/manager/0.log" Nov 03 19:54:15.753236 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:15.753203 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-2sbkn_5e3dc000-f9fd-43e0-b572-c14082077892/manager/0.log" Nov 03 19:54:16.024627 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:16.024598 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fvh8n_fd0b3242-3191-44dc-b805-14513b03baea/manager/0.log" Nov 03 19:54:17.109633 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:17.109587 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-r8rjd_1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e/manager/0.log" Nov 03 19:54:17.662073 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:17.662044 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pdm9s_aaceaf09-bcca-45f5-8b4d-2923f64e66e0/copy-manifests/0.log" Nov 03 19:54:17.741114 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:17.741080 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pdm9s_aaceaf09-bcca-45f5-8b4d-2923f64e66e0/manager/0.log" Nov 03 19:54:19.331012 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:19.330980 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-vqwqr_55fb4b93-d664-4763-9002-6b45c716ffbb/maas-api/0.log" Nov 03 19:54:23.528332 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:23.528304 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nlmfg_a1eb1eee-95b6-46aa-b05c-d37803323bf2/kuadrant-console-plugin/0.log" Nov 03 19:54:24.110543 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:24.110515 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9kbss_1788a2f1-9a31-481c-8fea-2f79d39a37a0/limitador/0.log" Nov 03 19:54:25.134719 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:25.134687 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-rpj78_ec4ed226-0f3b-429e-bda5-2348b4719de4/istio-proxy/0.log" Nov 03 19:54:26.947552 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:26.947519 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dxk85_679d0f01-dcb1-48f6-bcd4-cf627ecd8e95/istio-proxy/0.log" Nov 03 19:54:27.176195 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:27.176166 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-ftr65_8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4/istio-proxy/0.log" Nov 03 19:54:28.061309 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:28.061282 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wxx42_b5f6b782-1721-4ad3-bf95-63ee93f71a7c/istio-proxy/0.log" Nov 03 19:54:31.936436 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:54:31.936358 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:54:41.217593 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:41.217561 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xsfxn_8ede7475-dec7-4897-adc6-53c013df7362/manager/0.log" Nov 03 19:54:41.448523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:41.448489 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-2sbkn_5e3dc000-f9fd-43e0-b572-c14082077892/manager/0.log" Nov 03 19:54:41.725169 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:41.725141 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fvh8n_fd0b3242-3191-44dc-b805-14513b03baea/manager/0.log" Nov 03 19:54:42.744234 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:42.744199 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-r8rjd_1e8bdf57-5c5e-4bdf-84f5-5b58b700db5e/manager/0.log" Nov 03 19:54:43.293755 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:43.293727 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pdm9s_aaceaf09-bcca-45f5-8b4d-2923f64e66e0/copy-manifests/0.log" Nov 03 19:54:43.371762 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:43.371730 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pdm9s_aaceaf09-bcca-45f5-8b4d-2923f64e66e0/manager/0.log" Nov 03 19:54:44.650774 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:44.650731 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-vqwqr_55fb4b93-d664-4763-9002-6b45c716ffbb/maas-api/0.log" Nov 03 19:54:48.186758 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:48.186728 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nlmfg_a1eb1eee-95b6-46aa-b05c-d37803323bf2/kuadrant-console-plugin/0.log" Nov 03 19:54:48.722598 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:48.722566 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9kbss_1788a2f1-9a31-481c-8fea-2f79d39a37a0/limitador/0.log" Nov 03 19:54:49.549221 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:49.549189 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-rpj78_ec4ed226-0f3b-429e-bda5-2348b4719de4/istio-proxy/0.log" Nov 03 19:54:51.370495 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:51.370459 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dxk85_679d0f01-dcb1-48f6-bcd4-cf627ecd8e95/istio-proxy/0.log" Nov 03 19:54:51.594275 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:51.594238 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-ftr65_8e09e9d6-6ee8-4119-99d1-4da7dde7e8c4/istio-proxy/0.log" Nov 03 19:54:52.526252 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:54:52.526219 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wxx42_b5f6b782-1721-4ad3-bf95-63ee93f71a7c/istio-proxy/0.log" Nov 03 19:55:01.936869 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:55:01.936818 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:55:31.936562 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:55:31.936524 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:56:01.937265 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:56:01.937185 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:56:18.642626 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:18.642590 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x7248_8183d10b-0438-45e9-8e48-02c1cd3cc9f5/global-pull-secret-syncer/0.log" Nov 03 19:56:21.350900 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:21.350864 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xnctw_36e2360c-6ea0-40aa-835d-dc764716cd98/konnectivity-agent/0.log" Nov 03 19:56:21.709860 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:21.709787 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-236.us-west-2.compute.internal_0683633c577f920552f48984254eada9/haproxy/0.log" Nov 03 19:56:28.010609 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.010572 5209 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh"] Nov 03 19:56:28.010985 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.010932 5209 cpu_manager.go:414] "containerMap: removing container" podUID="d32a2317-7e48-452b-92ae-908d2571a215" containerName="maas-api" Nov 03 19:56:28.010985 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.010944 5209 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a2317-7e48-452b-92ae-908d2571a215" containerName="maas-api" Nov 03 19:56:28.011059 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.011020 5209 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a2317-7e48-452b-92ae-908d2571a215" containerName="maas-api" Nov 03 19:56:28.014020 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.014001 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.037284 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.037259 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-proc\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.037408 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.037307 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-lib-modules\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.037477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.037426 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87kh\" (UniqueName: \"kubernetes.io/projected/3991361a-9a44-40b9-9920-b3b3ef8ffa87-kube-api-access-x87kh\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.037477 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.037464 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-podres\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.037572 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.037544 5209 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-sys\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.117132 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.117088 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"kube-root-ca.crt\"" Nov 03 19:56:28.117313 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.117109 5209 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"openshift-service-ca.crt\"" Nov 03 19:56:28.117313 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.117226 5209 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t5k7x\"/\"default-dockercfg-c8qsb\"" Nov 03 19:56:28.118925 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.118905 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh"] Nov 03 19:56:28.138833 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138811 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-sys\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138870 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-proc\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138896 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-lib-modules\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138927 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87kh\" (UniqueName: \"kubernetes.io/projected/3991361a-9a44-40b9-9920-b3b3ef8ffa87-kube-api-access-x87kh\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138943 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-sys\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138950 5209 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-podres\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138991 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-podres\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139032 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.138993 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-proc\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.139407 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.139046 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3991361a-9a44-40b9-9920-b3b3ef8ffa87-lib-modules\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.200375 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.200348 5209 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87kh\" (UniqueName: \"kubernetes.io/projected/3991361a-9a44-40b9-9920-b3b3ef8ffa87-kube-api-access-x87kh\") pod \"perf-node-gather-daemonset-24bdh\" (UID: \"3991361a-9a44-40b9-9920-b3b3ef8ffa87\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.325588 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.325563 5209 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:28.380561 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.380539 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nlmfg_a1eb1eee-95b6-46aa-b05c-d37803323bf2/kuadrant-console-plugin/0.log" Nov 03 19:56:28.698867 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:28.698841 5209 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh"] Nov 03 19:56:28.699985 ip-10-0-133-236 kubenswrapper[5209]: W1103 19:56:28.699963 5209 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3991361a_9a44_40b9_9920_b3b3ef8ffa87.slice/crio-c2ef20c2024aa0f1898c3c6e71732779bc27fd727fa79d4ff32767210ba48047 WatchSource:0}: Error finding container c2ef20c2024aa0f1898c3c6e71732779bc27fd727fa79d4ff32767210ba48047: Status 404 returned error can't find the container with id c2ef20c2024aa0f1898c3c6e71732779bc27fd727fa79d4ff32767210ba48047 Nov 03 19:56:29.189878 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:29.189849 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9kbss_1788a2f1-9a31-481c-8fea-2f79d39a37a0/limitador/0.log" Nov 03 19:56:29.290350 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:29.290317 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" event={"ID":"3991361a-9a44-40b9-9920-b3b3ef8ffa87","Type":"ContainerStarted","Data":"fc8c5ec8da9a26e6ab1c9902332e0026dbc010167fb9b5dc1d7982fd57bda3a6"} Nov 03 19:56:29.290532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:29.290392 5209 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" event={"ID":"3991361a-9a44-40b9-9920-b3b3ef8ffa87","Type":"ContainerStarted","Data":"c2ef20c2024aa0f1898c3c6e71732779bc27fd727fa79d4ff32767210ba48047"} Nov 03 19:56:29.290532 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:29.290430 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:29.406731 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:29.406678 5209 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" podStartSLOduration=2.406664567 podStartE2EDuration="2.406664567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:29.405478926 +0000 UTC m=+4270.816579048" watchObservedRunningTime="2026-11-03 19:56:29.406664567 +0000 UTC m=+4270.817764690" Nov 03 19:56:31.936468 ip-10-0-133-236 kubenswrapper[5209]: E1103 19:56:31.936429 5209 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2c66j" containerName="extractor" Nov 03 19:56:32.154501 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:32.154471 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/alertmanager/0.log" Nov 03 19:56:32.556315 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:32.556277 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/config-reloader/0.log" Nov 03 19:56:32.962852 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:32.962819 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/kube-rbac-proxy-web/0.log" Nov 03 19:56:33.364592 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:33.364561 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/kube-rbac-proxy/0.log" Nov 03 19:56:33.765696 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:33.765604 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/kube-rbac-proxy-metric/0.log" Nov 03 19:56:34.166909 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:34.166882 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/prom-label-proxy/0.log" Nov 03 19:56:34.568413 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:34.568334 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6368e126-6e6c-43e6-a415-6ce9f87bfcb9/init-config-reloader/0.log" Nov 03 19:56:35.305373 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:35.305342 5209 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-24bdh" Nov 03 19:56:35.384406 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:35.384377 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vznsr_ad9cc12e-64ec-4536-8e8d-02f28afcd385/kube-state-metrics/0.log" Nov 03 19:56:35.776782 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:35.776755 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vznsr_ad9cc12e-64ec-4536-8e8d-02f28afcd385/kube-rbac-proxy-main/0.log" Nov 03 19:56:36.180423 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:36.180385 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vznsr_ad9cc12e-64ec-4536-8e8d-02f28afcd385/kube-rbac-proxy-self/0.log" Nov 03 19:56:36.585162 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:36.585131 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85778475ff-j2r6b_f815fdf2-7e68-4622-a30c-0e864a88b779/metrics-server/0.log" Nov 03 19:56:37.406140 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:37.406103 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4shpc_b32ce1b3-62ad-4273-8763-997294b83b96/node-exporter/0.log" Nov 03 19:56:37.816410 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:37.816331 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4shpc_b32ce1b3-62ad-4273-8763-997294b83b96/kube-rbac-proxy/0.log" Nov 03 19:56:38.218233 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:38.218209 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4shpc_b32ce1b3-62ad-4273-8763-997294b83b96/init-textfile/0.log" Nov 03 19:56:46.367700 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:46.367640 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c7dc9f994-29p67_9270f04e-1085-495a-9636-d4cba5958f5b/telemeter-client/0.log" Nov 03 19:56:46.769617 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:46.769539 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c7dc9f994-29p67_9270f04e-1085-495a-9636-d4cba5958f5b/reload/0.log" Nov 03 19:56:47.171523 ip-10-0-133-236 kubenswrapper[5209]: I1103 19:56:47.171494 5209 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c7dc9f994-29p67_9270f04e-1085-495a-9636-d4cba5958f5b/kube-rbac-proxy/0.log"